BalaBit Blog

I had to think long and hard about this interpretation, as it wasn’t certain whether the last revelation was highlighting something that already happend, or whether it was a prophecy, pointing forward. I know now that it was a bit of both. As one could easily figure out, the Revelation is about syslog-ng: it is about the burden of keeping records – logs -, about scalability, and methods to keep up with the demand.

And this is why I was torn whether it’s a record of events, or a prophecy: did syslog-ng achieve higher levels of scalability recently? Yes, it did! Bazsi‘s threading work is in syslog-ng 3.3 branch, and it improves performance quite spectacularly in most cases. But is that all? Would threading be as much an improvement as papyrus would be over clay tables?

I don’t think so. It’s a logical step forward, especially so, because there was at least one driver using a worker thread before: the idea is not new. Yet, it’s a significant improvement nevertheless, but it is also something that could be foreseen, something Master Wall wouldn’t need to highlight. Don’t get me wrong: the way threading was implemented in syslog-ng is brilliant, but that has been a long planned feature, something we already expected to see. Therefore, the Revelation must be talking about something else. Or perhaps not: it wouldn’t be suprising if Master Wall thought of everything, and in his great wisdom, he’d foresee not only the inevitable, but the enlightening improvement aswell, and hint at both at the same time.

Thinking this further, one has to wonder how exactly logging can be improved. Speed is certainly a factor, but that’s neither new or suprising. Scalability is in the same shoes, so is support for various new protocols and formats. Interesting, welcomed, useful – but suprising? Not really. Lets have a look at the history of logging, maybe that’ll help us reach enlightenment!

In the beginning, someone had the great idea to write the current state of his application to the screen, and saw that this was good, except there was no way he could keep up with it, no way to search, and no persistent storage. So he started to write into files, and liked it. He could search it, he could store it, back it up, and all was well. Up until he had to do this routine for the tenth application, and the whole process became tedious: many files, scattered all around, all a little bit different. It dawned upon him: a central solution is needed: a dedicated program to accept log messages, and write them to files! Astounding revelation!

And so it happened, that the first syslogd was born. But it too, suffered from the lack of design and foresight: the format was not well defined, vague, and incomplete: there was no way to structure log data properly, which made searching and log-based alerting a black art, practiced only by the greatest of wizards with long beards and strange mascots on their t-shirts.

Then, a new protocol – loosely based on the old ad-hoc one – was developed, with clear instructions on how one can structure data, and people liked it. For a while, at least… Then they realized that this doesn’t solve the problem of finding the information, as they’re still stored in flat files, and searching in those is slow at best. Nevermind the other serious flaws of such a setup.

Earlier, someone had the great idea of putting log messages into a searchable database, and the crowd cherished the idea: they began to mindlessly put everything inside databases, and it worked, for a while. But then structured data entered the picture, and the database designers were baffled: how could one possibly store all the data in the conventional, table-based approach? There’s so many of them! So unpredictable!

And they did not store structured data, but stored selected properties, and one could only use those to search and build alerts upon. And while this scaled well, it didn’t make anyone happy, quite the contrary.

And this is where Master Wall’s brilliant foresight comes back into the picture: Why should we continue to use the old methods of beating everything into a format it doesn’t naturally lend itself to? Why not do the obvious, and store the logs in a flexible, structured format?

Why are so many people obsessed with logging messages to SQL, when SQL databases rarely, if ever, lend themselves to fairly ad-hoc storage? They have a rigid schema, one cannot deviate from, and thus, the solution works as long as the messages conform, but as soon as they do contain structured data the database wasn’t prepared for, it’s a lost cause.

But alas, as Master Wall predicted: there is a solution! The various document stores started to receive well deserved attention recently, and one of them even proclaims that the end of the one-size-fits-all paradigm is over, acknowledges that for different tasks, different tools are better. This particular database is MongoDB, and it is a document store: it can not only store structured data, but it has a query mechanism that allows for fast retrieval and considerable searching possibilities (even more so if one goes the long way of map/reduce).

Why aren’t we using this to our benefit then? Having no schema means we can store arbitrary data! We do have common fields, but all the rest could be stored along, and could later be queried.

That’s a good question, and the answer is simple: up until recently, we couldn’t, syslog-ng did not support storing log messages in MongoDB. But with the upcoming syslog-ng OSE 3.3 release, that is going to change, the afmongodb destination driver has been merged. While there’s still work left to do, a few bits and pieces still left to merge, just imagine the sheer amount of joy the Scribes of Old would’ve felt knowing these news: You will be able to write not only with two hands, but with your feet and artifical appendages, you will be able to store record in ways flexible enough to store everything! No need to invent workarounds, new symbols, new methods to search: it’s all there, well prepared.

That, ladies and gentlemen, is syslog-ng OSE 3.3, with threading and afmongodb. This is how the Scribes interpreted Master Wall’s question, a bright future ahead!