Centralized logging of events has been an important part of the IT infrastructure for many years. It is more convenient to browse logs in a central location rather than viewing them on individual machines. Central storage is also more secure. Even if logs stored locally are altered or removed, one can still check the logs on the central log server. Compliance with different regulations also makes central logging necessary. (This is an updated version of my previous syslog-ng web gui blog.)

System administrators often prefer to use the command line. Utilities such as grep and awk are powerful tools but complex queries can be completed much faster with a SQL-based web interface. In the case of large amounts of messages, a web-based SQL solution is not just convenient, it’s a necessity. With thousands of incoming messages per second, the indexes of log databases still give Google-like response times even for the most complex queries, while traditional text based tools are not able to scale as efficiently.

Web Graphical User Interfaces (GUI) for syslog-ng range from simple scripts to browse logs to cloud-based logging for enterprise-level applications. All of these have different strengths and weaknesses, and target different usage scenarios. Some of the most popular GUIs used with syslog-ng are described below.

Loggly

Loggly, a cloud-based service, does not require a local software installation. First an account needs to be created at Loggly. Once logged in, a wizard interface provides any information to get started in a few minutes. The necessary syslog-ng configuration snippet is ready for copy & paste after a few clicks.

Loggly is a commercial service where payment is based on the daily amount of logs and how long the logs need to be stored. There is also a free developer account available, which has all the features of the paid edition, but has strict limits on the amount of logs and storage time.

The web interface is clean, easy to use. It has a command line interface, which will look familiar to UNIX and network operators. Still, it is very powerful, as it can query or graph data in an instant, and running modified queries is well supported by the command line history.

It is an ideal solution for those wanting to set up central logging quickly, without the need to maintain yet another local system.

URL: http://www.loggly.com/

There are two more cloud based services, which we did not investigate yet in detail:

• http://logentries.com/ which has many interesting technologies, including functions for major programming languages and platforms to send logs to their service
• https://papertrailapp.com/ which can send alerts back to applications over http

Loganalyzer

Loganalyzer is a lightweight PHP application to browse and query logs stored in a MySQL database. Not originally written for syslog-ng, it can be easily configured for use with syslog-ng with some minor limitations (there are no separate fields for program name and process IDs). As it uses MySQL for log storage and indexing, it does not scale well. Using simple inserts with MySQL limits input to just a few hundred messages per second, and the lack of external indexing makes even simple queries on millions of messages cumbersome. Loganalyzer does have some minimal log management built in as database tables can be cleaned from the admin menu.

Logalazer is suited to users without many log messages to process and are looking for a completely free web GUI solution. Commercial support is also available.

URL: http://loganalyzer.adiscon.com/

LogStash

Logstash is a tool to collect, filter and display logs. It can collect logs from many sources, including syslog-ng, filter and store them in a database that can be searched from a web interface. It can also output logs in various formats.

Still a 1.0 version, with the limitations often found in first versions, it is, however, one of the easiest to get started with it. One only needs to download a single file, edit a simple configuration file, to begin collecting logs from files or a central syslog server. LogStash’s web interface is simple and easy to use but search history and stored searches are not currently supported.

Using Elasticsearch as a backend, LogStash is better able to scale than simple, MySQL based solutions. Log management is not yet available from the web interface; old logs are not automatically erased and must be deleted manually by accessing elasticsearch directly.

LogStash is ideal for users who want to setup a scalable system in a matter of minutes and browse their logs immediately. It can be fine tuned and extended while becoming familiar with the application. In advanced configurations, the web interface and the databases can be separated to different machines and elasticsearch can further be clustered to multiple machines.

URL: http://logstash.net/

Logzilla

Logzilla is the commercial reincarnation of one of the oldest syslog-ng web GUIs: php-syslog-ng. It provides a familiar interface for users of its predecessor, but also includes many new features. The user interface supports Cisco Mnemonics, extended graphing capabilities and e-mail alerts. Behind the scenes LDAP integration, message de-duplication were added, and indexing for quick searching on large datasets.

Logzilla has a flexible pricing structure based on several factors including features, number of logged hosts and number of messages. A free edition with limited capabilities, suited to smaller networks, is also available.

There is a number of different ways to get started. The easiest way is to download a vmware appliance, start it, paste a license to the web interface and start adding clients. If the expected log volume is higher, it is recommended to avoid virtualisation for better performance. This requires many more steps, but results in much better scalability.

Logzilla is an ideal solution for a NOC full of Cisco devices. It can handle a large amount of messages thanks to its optimized MySQL access and sphinx.

URL: http://www.logzilla.pro/

ELSA – Enterprise Log Search and Archive

Enterprise log search and archive (ELSA) is a brand new centralized syslog framework with syslog-ng at its heart. It is the first larger project outside of BalaBit utilizing the power of its patterndb log classification tool.

The ELSA architecture is designed to work at high, continuous incoming message rate and to bear even higher load peaks for hours. Instead of regular expressions, used by other solutions, it utilizes patterndb, which is more powerful and needs less resources. For example, one can narrow down a search in which a given IP address is the target of an operation, rather than perform a general IP address search. Sysadmins’ work is made easier by several features including a tab based UI to run related queries in parallel, scheduled searches, and easy to build queries using menus.

For better scalability, the web interface, the database and the indexers can be distributed to multiple machines. This distributed configuration ensures that even an extreme amount of log messages can be searched with Google-like response times.

ELSA is an ideal solution for organizations with a massive amount of logs that can afford to invest the time for installation and do not require in depth documentation.

syslog-ng store box (SSB) GUI

SSB is a log management appliance to collect, classify, organize, and securely store log messages for enterprises who operate log infrastructure for compliance and maintenance reasons. As the official GUI for syslog-ng, the SSB GUI is fully able to utilize all the syslog-ng Open Source Edition and Premium Edition features. The SSB web-based GUI supports customizable user role separation and fine-tuned access control to sensitive logs. The web interface is accessible via a network interface dedicated to the management traffic. This management interface is also used for backups, sending alerts, and other administrative traffic. All configuration changes are automatically logged, simplifying audits.

■ Read more about BalaBit syslog-ng and SSB products

■ Request evaluation version

■ Request callback