Log Management with Log4net and Microsoft Azure
Determining how to efficiently manage logs for your large scale application(s) can be a daunting task. This is particularly important when running PaaS services, such as App Services, Web Jobs, and Azure Functions, where real-time access to application log files is not easily accessible. One effective solution leverages the log4net framework and a number of Microsoft Azure services for a surprisingly intuitive and scalable architecture. Most logging solutions allow for similar customizations. NLog is another popular logging framework that can also be customized to perform the same function. The following demo is done in C#.
How to Create a Custom Appender using Log4Net:
1) Create an Azure Service Bus Queue.
- This step requires that you have an active Microsoft Azure subscription. You will have to first create a new Service Bus in the Azure portal. Azure Service Bus is a scalable and robust platform for log management. It provides one central place where you can pool your messages from one or many applications. Azure supports queue depth scaling to allow for automatic scaling when the queue gets backed up.
- After you navigate to your new Service Bus you will be prompted with a series of controls. If you select the “Queues” control on the left of the navigation and then the “+ Queue” control on the top of the menu, you will be able to create a queue for your Service Bus (with a number of configurable settings).
- For this example I named the queue “test_queue1”. Next, you will have to create a shared access policy (to allow access for sending, listening, or both) for the queue. Once this is complete the Azure portal will produce a connection string for you. You will need this connection string in order to…
2) Configure your Custom Appender
- You can add this ConnectionString to the ConnectionString section of the config file from the application that you want to log from as shown below:
- You will have to add log4net to your project (available as a Nuget package). Next, you will have to add some configuration settings into your project.
- A log4net config section will have to be added into the config file of the application from which you wish to log. It will look something like this:
- The type attribute in the appender setting refers to the location of the custom appender that we will be writing in the next step. The queueConnectionName and applicationName settings will be referenced in the next step as well. More information on the log4net config settings can be found here.
3) Create a Custom Log4net Appender
- For this step we will create custom log4net appender class “AzureLoggingAppender”. This class will inherit from the AppenderSkeleton class (part of the log4net.Appender namespace). The class will have a string attribute called QueueConnectionName (“LoggingQueue” for our example), and an ApplicationName to specify the application where the log is coming from. Both of these are passed in through the highlighted config settings above. The next step is to override the Append of the AppenderSkeleton class. This method takes a log4net LoggingEvent object as a parameter and then retrieves the Queue Connection String by using the Queue Connection Name.
- Using that connection string, and the queue name that we specified in Step One, we can create a new QueueClient (Microsoft.AzureLogging.Messages namespace), as shown below:
- You can then create a LogMessage class with whatever data you choose to populate your logs. . Our example will have a simple LogMessage with 4 fields populated with data from the logging event or passed in to the AzureLoggingAppender.
- The next step is to send the log to the Service Bus queue. We create a new BrokeredMessage (Microsoft.ServiceBus.Messages namespace) using the LogMessage class and send it to the Service Bus queue using the QueueClient we created earlier.
- Once this completes, the message will be delivered to your Azure Service Bus queue.
4) Add a Reference to the New Appender
- Add a reference to the newly created library in the web.config or app.config of your project. The “type” attribute of the appender setting shown in step 2 will have to be the fully qualified location of the new appender. In our example the appender class is located at “Sample.Logging.AzureLoggingAppender”.
One of the biggest advantages of using Service Bus to manage your logs is the versatility you have once the messages are in the proper queue. Once the appender is in place, all of your applications will have the ability to post their logs to the central queue with very little setup. You can then have multiple applications share the same appender and simply submit the different application names through configuration.
One option would be to use an Azure Web Job to subscribe to the queue and write the messages to a shared database, send email notifications, both, or perform numerous other options. Similar features can be built using Azure Functions or even Logic apps. As the above images illustrate, Log4net and Azure Service Bus provide a powerful and scalable solution that should be considered for all developers looking to simplify their log management experience.
Example Integration: Microsoft Teams
With a few lines of code you can easily log your urgent messages to a Teams channel and immediately start to collaborate on issues.
To learn more on how Tallan’s Cloud technologists can help your organization enhance productivity, streamline costs, and simplify the management of your data within the cloud, CLICK HERE.