Microsoft AD DNS Debug Logging Format Error

I have been doing some experimentation in hopes of exporting and analyzing Microsoft AD DNS debug logs. The project goal is to export the relevant data to a Graylog analyzer with an Elasticsearch back-end. The basic steps are trivial. The biggest issue faced is being able to correctly parse the DNS debug logs so the data can be parsed and sent to the log analyzer. We are using NXLOG to ship the data to Graylog. The problems arise in the inconsistent format used by Microsoft in the log formatting.

Here are just a few of the problems:

  1. The log file contains the Field Definitions displayed in the first 29 rows of the log file. There is no way to configure the DNS server to omit this data from the log file
  2. The field lengths for data in each field is inconsistent with some variable length fields intermingled with fixed length fields.
  3. Field structure is dependent upon the logging options chosen
  4. The last field, the actual query question, it formatted without the . notation and contains (some number) indicating the number of spaces until the next . in the query name
  5. The log is written with every other row being blank
  6. If the “details” option is picked, the details of each query is inserted into the log immediately following the log line
  7. The logs don’t rotate by time, only size
  8. Restarting the DNS service causes the log file to re-initialize
  9. Enabling “Log unmatched incoming response packets” introduces an obvious bug in the formatting of each log row that contains an alert for an unmatched packet. The LF/CR is completely missing from the end of the line indicating the unmatched response so two log lines get jammed together. This is an outright bug in the MS logging code.

I point all this out simply to indicate how difficult it is to properly parse and rely upon the integrity of the native DNS debug log produced by AD DNS server. DNS logging and proper analysis is critical in enforcing excellent network security. Microsoft makes it quite difficult by not incorporating better logging mechanisms into their AD DNS server.

I am working on a project that will hopefully remedy that problem.

I did notice in the Graylog Marketplace, there is a Content Pack listed called Windows DNS Content Pack. I did attempt to use this and it does a decent job of adding the proper Grok patterns and Extractors along with a decent dashboard. The output data was not very well formatted (a limitation of Grok) and proved less useful than I hoped.

As of 10-08-2015, I have solved what I set out to do. I have NXLog sending data from my AD servers. I was able to build a Graylog Plugin using Java to act as an input and parser for my DNS Debug data. I will write a separate post on this shortly and post the plugin on my Github site.

2 Comments

  1. Hi, your articles on how to do this are awesome. I have been trying to find a good solution to analyze DNS logs (without something like splunk) for a while now, and you’ve convinced me that graylog is the way to go. I’ve setup the CentOS VM, installed graylog and NXlog, but I was wondering where I could find the plugin you wrote in order to parse them correctly?

    1. Austin,

      Thanks for the feedback. I’ll put the Java plugin up shortly. Ultimately, it will be on Github and on the Graylog site. I’ll hit you up when it is ready and posted. You can still have Graylog ingest the data from NXLog but the fields won’t be pretty and you won’t have the ability to do nice searches. Talk to you soon.

Leave a Reply

Your email address will not be published. Required fields are marked *