Posts Tagged ‘mongodb’
Just before christmas I’ve released syslog-ng 3.4.0beta1, hopefully the last before the final release of syslog-ng 3.4. You can find the list of changes since 3.4.0alpha3 here.
Here’s the teaser for those wondering what 3.4 can do for them in addition to 3.3:
- Junctions, channels and the improved configuration format allows specifying log processing rules at even more flexibility.
- Full json support by the introduction of json-parser() and enhancements to $(format-json) template function.
- Support for the amqp() destination that implements support for the AMQP queueing protocol was added.
- MongoDB improvements to support replicasets, UNIX domain sockets. The performance was also improved by using insert operations instead of upserts.
- Added support for sending emails via the smtp() destination.
- Allow huge messages, instead of the old limits of 256k per message and 64k per value limits, the limits are 4GB for both.
- Added support for parsing the syslog message format after the initial reception. This can be used to “fix” messages before it actually reaches the syslog parser stuff.
- Native support for systemd.
- Demand loadable plugins to avoid having to explicitly write “@module” statements. This makes writing syslog-ng.conf files easier.
- A number of new template functions, like $(uuid), $(hash) and so on.
- A number of new macros $LOGHOST, $C_DATE and friends.
- A number of new parsers in db-parser, @PCRE@, @EMAIL@, @SET@, …
- Added rewrite operations to change message tags.
- Improved value-pairs expression support that allows specifying
More details can be found in the individual release announcements:
- 3.4.0alpha1: http://bazsi.blogs.balabit.com/2012/03/first-alpha-release-of-syslog-ng-3-4-published/
- 3.4.0alpha2: http://lists.balabit.hu/pipermail/syslog-ng/2012-May/018746.html
- 3.4.0alpha3: http://lists.balabit.hu/pipermail/syslog-ng-announce/2012-June/000144.html
As with all versions of syslog-ng, this wouldn’t have been possible without the help of the syslog-ng community. The role of the community is increasing with release-to-release, larger and larger features are contributed outside BalaBit and the number of contributors is increasing steadily too. I’d like to grab this opportunity to say thanks for everyone involved. Help is welcome and appreciated, be that code, documentation, a description of a use-case or simply just feedback. Thank you.
Stay tuned for the final version!
As usual, in recent weeks, I’ve been too busy to write, but I’m making up for it now. Over the past months, there have been a few very significant changes, some were already announced, some were not, but this is a great opportunity to celebrate all of them again. The most important thing, from my perspective is that from the 15th of September this year, my role within BalaBit shifts slightly, and I will only work half-time at support (as opposed to the current ~80%), in the other half, I’ll be a syslog-ng OSE developer, working on reducing the gap between the current syslog-ng PE core and OSE, among other things. I’m already hyped up beyond imagination, even though I realize that this only means my TODO list will expand into infinity within two hours. Nevertheless, great things are ahead!
Again, on the syslog-ng front, I released syslog-ng 3.3.6, started working on a feature that makes it possible to emit structured JSON, and allow us to move from upserts to inserts in the MongoDB destination. This was originally planned to be a GSoC project, but sadly it fell to me in the end. However, I promised to deliver the feature by the end of GSoC no matter what, and I prefer to keep my promises, and so I did.
Elsewhere, I’ve spent a lot of time exploring various technologies that I wish to use in the near future, built proof of concept demos for about a dozen ideas I’ve had – some of these will likely be showcased sometime soon.
On the other hand, it has not all been joy and happiness: after much consideration, I decided that the interest I had in CEE and lumberjack is no more, the reasons I became interested in are now void, therefore I will stop following and promoting both, because neither will fulfill my needs in the future. As such, maintainance of the libumberlog library I wrote earlier this year will be handed over to a collegue of mine in the next coming days.
In September, I will start university again, and this time, I made sure I go somewhere that I will not only be able to finish while maintaining my sanity (what little is left of it, anyway), but attend courses that will benefit me greatly in the long run, much more than previous attempts had. This, combined with being allowed to work half-time on syslog-ng OSE rearmed my fading enthusiasm, and that instantly swept away a lot of worries that troubled me in the past months.
As for what surprises September holds, I’ll keep you posted! I promise there will be a couple of very interesting demos presented!
Dear syslog-ng users,
This is the 13th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news.
Your feedback and news tips about the next issue is welcome at
Have you tried to add custom information to log messages, fix mis-formatted logs or anonymize logs?
The next long-time-supported release of SSB version 3 LTS is about to be released. This release includes a switch to 64-bit architecture, a huge performance improvement in the indexing/searching feature for a large number of message and search patterns and a couple of new features, too. The following post plans to introduce those new features to you. The updated User’s Manual will contain a detailed description of them — this post is written more to serve as teaser and to highlight some of the use cases we had in mind when we’d planned the features, and, of course, to ask for your feedback about them.
- Admin guide
syslog-ng participates in GSoC
This year syslog-ng participates in GSoC under the umbrella of openSUSE. We have one student accepted, who will work on syslog-ng’s mongodb destination.
One of the major reasons to update to 3.3 other than threading is MongoDB. It allows great flexibility when using patterns and storing parsed data from logs. The syslog-ng documentation covers most information necessary to use MongoDB, but this HowTo compiles all these into a single document and extends it with features from the upcoming syslog-ng 3.4 version.
OTHER SHORT NEWS
- There is now an open source LogStore reader work in progress
- syslog-ng 3.4 alpha packages are now available in the FreeBSD ports and also for openSUSE
- syslog-ng and ELSA were presented at LOADays.
There are a thousand and more reasons I believe in Free Software, but the most important part for me is that free software allows me to tinker, to play and have fun. Even better, it allows everyone to do the same, and presents opportunities to play together and make a piece of software even better than it already is, for everyone’s benefit. Free software makes things like Goggle Summer of Code possible, where students can work on a piece of software, be part of the larger community, and get a gentle introduction to this beautiful world.
It is a very satisfying feeling when one sees his own work being developed further by others. For me, this means that my work was useful for not only me, but others aswell, and even better: it was found interesting enough to improve upon! For this reason, I’m looking forward to the summer, to see what comes out of it, and I’m eternally thankful for the good folk at OpenSuSE, who gave us this opportunity to participate in this year’s GSoC, and allowed us a slot for this project.
And what this project is about, one might wonder? It’s about one of the most often requested functionality of the MongoDB destination driver: make it faster, better and more scalable! Right now, the driver uses something called an upsert, which is a kind of insert-or-update functionality. The reason it does so, is because syslog-ng stores its internal name-value pairs as a flat list, not as a tree, and using upsert allows the driver to use the keys as-is, as a flat structure (thanks to upsert being able to deconstruct a flat JSON dotted-notation into proper structures). The major advantage of this is that syslog-ng doesn’t need to do any expensive transformation. The downside is that upserts are inherently slow, and cannot be done in bulk, and place a higher load on the server than one would wish to.
The GSoC project is about fixing this situation: its goal is to teach syslog-ng to construct a tree out of flat name-value pairs, and provide an API not only for MongoDB, but for other drivers that need this (the JSON formatter comes to mind). Once this is done, the driver can be switched to use inserts, and reap all the benefits: higher speed, less load on the server, and an opportunity to do bulk inserts, which reduce the load even further.
This is a hard task, yet, a very useful one, not only for the MongoDB destination driver, but any other that may wish to deal with structured data in the future. But not only that! It is – I believe – also a great opportunity to become part of the syslog-ng community, to learn a lot about an important piece of a system’s infrastructure: the syslog daemon (and the best one of its kind at that). Even more, Eun Kyung will also have to work with MongoDB, to make sure the driver performs as well as we expect it to. There’s a lot to learn and explore, a lot to improve upon – this is the beauty of free software.
And this is why I’m looking forward to see the results, and hear how Eun found the summer to be. I sincerely hope – nay, know – it will be a fun and rewarding experience for all of us.
I had to think long and hard about this interpretation, as it wasn’t certain whether the last revelation was highlighting something that already happend, or whether it was a prophecy, pointing forward. I know now that it was a bit of both. As one could easily figure out, the Revelation is about syslog-ng: it is about the burden of keeping records – logs -, about scalability, and methods to keep up with the demand.
And this is why I was torn whether it’s a record of events, or a prophecy: did syslog-ng achieve higher levels of scalability recently? Yes, it did! Bazsi‘s threading work is in syslog-ng 3.3 branch, and it improves performance quite spectacularly in most cases. But is that all? Would threading be as much an improvement as papyrus would be over clay tables?
I don’t think so. It’s a logical step forward, especially so, because there was at least one driver using a worker thread before: the idea is not new. Yet, it’s a significant improvement nevertheless, but it is also something that could be foreseen, something Master Wall wouldn’t need to highlight. Don’t get me wrong: the way threading was implemented in syslog-ng is brilliant, but that has been a long planned feature, something we already expected to see. Therefore, the Revelation must be talking about something else. Or perhaps not: it wouldn’t be suprising if Master Wall thought of everything, and in his great wisdom, he’d foresee not only the inevitable, but the enlightening improvement aswell, and hint at both at the same time.
Thinking this further, one has to wonder how exactly logging can be improved. Speed is certainly a factor, but that’s neither new or suprising. Scalability is in the same shoes, so is support for various new protocols and formats. Interesting, welcomed, useful – but suprising? Not really. Lets have a look at the history of logging, maybe that’ll help us reach enlightenment!
In the beginning, someone had the great idea to write the current state of his application to the screen, and saw that this was good, except there was no way he could keep up with it, no way to search, and no persistent storage. So he started to write into files, and liked it. He could search it, he could store it, back it up, and all was well. Up until he had to do this routine for the tenth application, and the whole process became tedious: many files, scattered all around, all a little bit different. It dawned upon him: a central solution is needed: a dedicated program to accept log messages, and write them to files! Astounding revelation!
And so it happened, that the first syslogd was born. But it too, suffered from the lack of design and foresight: the format was not well defined, vague, and incomplete: there was no way to structure log data properly, which made searching and log-based alerting a black art, practiced only by the greatest of wizards with long beards and strange mascots on their t-shirts.
Then, a new protocol – loosely based on the old ad-hoc one – was developed, with clear instructions on how one can structure data, and people liked it. For a while, at least… Then they realized that this doesn’t solve the problem of finding the information, as they’re still stored in flat files, and searching in those is slow at best. Nevermind the other serious flaws of such a setup.
Earlier, someone had the great idea of putting log messages into a searchable database, and the crowd cherished the idea: they began to mindlessly put everything inside databases, and it worked, for a while. But then structured data entered the picture, and the database designers were baffled: how could one possibly store all the data in the conventional, table-based approach? There’s so many of them! So unpredictable!
And they did not store structured data, but stored selected properties, and one could only use those to search and build alerts upon. And while this scaled well, it didn’t make anyone happy, quite the contrary.
And this is where Master Wall’s brilliant foresight comes back into the picture: Why should we continue to use the old methods of beating everything into a format it doesn’t naturally lend itself to? Why not do the obvious, and store the logs in a flexible, structured format?
Why are so many people obsessed with logging messages to SQL, when SQL databases rarely, if ever, lend themselves to fairly ad-hoc storage? They have a rigid schema, one cannot deviate from, and thus, the solution works as long as the messages conform, but as soon as they do contain structured data the database wasn’t prepared for, it’s a lost cause.
But alas, as Master Wall predicted: there is a solution! The various document stores started to receive well deserved attention recently, and one of them even proclaims that the end of the one-size-fits-all paradigm is over, acknowledges that for different tasks, different tools are better. This particular database is MongoDB, and it is a document store: it can not only store structured data, but it has a query mechanism that allows for fast retrieval and considerable searching possibilities (even more so if one goes the long way of map/reduce).
Why aren’t we using this to our benefit then? Having no schema means we can store arbitrary data! We do have common fields, but all the rest could be stored along, and could later be queried.
That’s a good question, and the answer is simple: up until recently, we couldn’t, syslog-ng did not support storing log messages in MongoDB. But with the upcoming syslog-ng OSE 3.3 release, that is going to change, the afmongodb destination driver has been merged. While there’s still work left to do, a few bits and pieces still left to merge, just imagine the sheer amount of joy the Scribes of Old would’ve felt knowing these news: You will be able to write not only with two hands, but with your feet and artifical appendages, you will be able to store record in ways flexible enough to store everything! No need to invent workarounds, new symbols, new methods to search: it’s all there, well prepared.
That, ladies and gentlemen, is syslog-ng OSE 3.3, with threading and afmongodb. This is how the Scribes interpreted Master Wall’s question, a bright future ahead!
Update: The driver has a homepage of its own at http://asylum.madhouse-project.org/projects/syslog-ng/mongodb/
Though I had no chance to look at it yet, Algernon has posted a MongoDB destination driver for syslog-ng. I can’t wait to have a closer look at it, hopefully I get a chance in the coming days, but until then be sure to check it out. The announcement went to the syslog-ng mailing list, here’s a direct link: