Best XML format for log events in terms of tool support for data mining and visualization?

Question

We want to be able to create log files from our Java application which is suited for later processing by tools to help investigate bugs and gather performance statistics.

Currently we use the traditional "log stuff which may or may not be flattened into text form and appended to a log file", but this works the best for small amounts of information read by a human.

After careful consideration the best bet has been to store the log events as XML snippets in text files (which is then treated like any other log file), and then download them to the machine with the appropriate tool for post processing.

I'd like to use as widely supported an XML format as possible, and right now I am in the "research-then-make-decision" phase. I'd appreciate any help both in terms of XML format and tools and I'd be happy to write glue code to get what I need.

What I've found so far:

log4j XML format: Supported by chainsaw and Vigilog. Lilith XML format: Supported by Lilith

Uninvestigated tools:

Microsoft Log Parser: Apparently supports XML. OS X log viewer:

plus there is a lot of tools on http://www.loganalysis.org/sections/parsing/generic-log-parsers/

Any suggestions?

Answer 1

Unfortunately, I can't give you the answer you are looking for, but I would like to warn you of something to consider when logging to XML. For example:

<log>
 <msg level="info">I'm a log message</msg>
 <msg level="info">I'm another message</msg>
 <!-- maybe you won't even get here -->
 <msg level="fatal">My server just ate a flaming death
In the above snippet of a potential XML log you can see the biggest drawback of logging to XML. When a catastrophic failure happens, your log format becomes broken because it requires closing tags. However, if you are using a program that parses your primary log output, this shouldn't be too much of a problem.