Secure central log server with syslog-ng and SSH


I chose to port forward syslog-ng traffic over SSH versus configuring syslog-ng over SSL due to the fact that I already had a port open on my servers for SSH and due to the fact that SSL attempts to make the connection with unencrypted channels and SSH encrypts all communications to and from the client and server. Here is an excellent how to for tunneling over SSH: another helpful link was this:

Setup the tunnel

The first step is to get the SSH tunnel set up between the two machines. My personal preference is to originate the SSH tunnel on my central “loghost” machine at the primary site, and have it connect to the machine at the “remote” site that I want to get logs from. Typically this involves drilling out through the firewall at the primary site– often the site’s default firewall rules will allow this connection without any reconfiguration– and allowing the connection “inward” through the firewall at the remote end, which usually requires some firewall ruleset tweaks on the remote site’s firewall.

However, since we want the remote log server to be sending logs back to the central loghost at the primary site, we need to use a reverse tunnel (that’s the “-R” option on the SSH command line) to get things working properly. This is actually one of very few places where I find reverse tunnels to be useful. Figure A, below, shows a high-level picture of how the traffic is flowing in this design.

We need to make sure that the SSH session and tunnel are set up automatically when the central log host boots. If the SSH session dies for some reason (intermittent network outage, system administration “accident”, etc) we’d also like the connection to be re-established as quickly as possible. In situations like this, I like to have the init process fire off the SSH connection with a line like this in /etc/inittab:

log1:3:respawn:/usr/bin/ssh -nNTx
   >/dev/null 2>&1

The example above must appear as a single long line in /etc/inittab— I’ve just broken it onto multiple lines for clarity.

Let’s examine the SSH command line first. The “-R” on the second line of the example sets up the reverse tunnel from 514/tcp on the remote server to ““– in other words, port 514/tcp on the central loghost machine. While it seems natural to use 514/tcp for Syslog-NG logging, you have to remember that 514/tcp is the reserved port for the Unix rlogin/rsh service so you’re going to run into a port conflict if you still have these services enabled. I generally turn off unencrypted network protocols like telnet, FTP, rlogin/rsh/rcp on my servers and use SSH instead, so it’s not an issue for me, but you can run this tunnel over any free ports if there is a conflict at your site.

As for the other SSH command line options above, the “-n” flag tells SSH to associate the standard input with /dev/null. There won’t be any command line input since we’re essentially going to be running the SSH client as a “daemon” via init. As you can see at the end of the command line in the example, we’re also sending the standard output and standard error to /dev/null as well (“… >/dev/null 2>&1“). Since we’re never going to be issuing remote commands via this SSH connection (we only care about the tunnel), the “-N” option to SSH tells the SSH client to only set up the tunnel and to not bother preparing a command stream for issuing commands on the remote system, while “-T” says to not bother allocating a pseudo-tty on the remote system. The “-x” option disables X11 forwarding, just as a defense-in-depth gesture.

Turning our attention to the rest of the /etc/initab entry, the first field (“log1“) is just an identifier for this entry in the inittab file. These identifiers can be any sequence of 2-4 alphanumeric characters; the only requirement is that they be unique from all other identifiers used in the file. I’ve chosen “log1” here because it’s usually the case that I have multiple SSH tunnels set up to different remote log sources, and I typically name the inittab entries “log1“, “log2“, etc. The second field in the inittab file (“3“) is the run level where this entry should be fired. Make sure to start this SSH process after the network interfaces have been initialized but before the Syslog-NG daemon is started.

The “respawn” option in the third field is the reason I like to use init for spawning processes like this. When the “respawn” option is enabled, the init process will automatically fire off a new SSH process if the old one dies for any reason. In other words, init acts a like a “watchdog” type daemon and makes sure that the SSH tunnel is always up and running. This is an extremely useful technique, but one that a lot of system admins seem to have forgotten.

Once you’ve got your inittab entry all set up, HUP the init process (“kill -HUP 1“). This should cause the init process to re-read the inittab file and spawn the SSH connection. You should be able to verify that the SSH client is running with the ps command and verify the existence of the tunnel using netstat. Once you’ve got all that working, it’s time to turn our attention to configuring Syslog-NG.

Test the connection via command line and make sure you are not prompted for a password.

server # /usr/bin/ssh -nNTx -R 1514:

Installing Syslog-NG

Download the appropriate .rpm from for your system

Install with rpm -ivh syslog-ng.*.rpm

Default syslog-ng configuration

You can view the locations of all the files and binaries installed via the rpm package: rpm -ql syslog-ng

The good thing about and rpm package is that it installs init scripts and logrotate config files for you.


  • Options – used to configure global options
  • Source – One or more source sections are used to control where messages may originate
  • Destination – One or more destination sections declare where messages should be sent
  • Filter – Declare filters that should be applied to messages.
  • Log – Include one or more sources, one or more filters, and one or more destinations

The default syslog-ng.conf file that comes with RHEL5 is configured to perform like the syslogd service that is installed by default.


There is one source configured for the system called s_sys. It is defined here:

source s_sys {
file (“/proc/kmsg” log_prefix(“kernel: “));
unix-stream (“/dev/log”);
#udp(ip( port(514));

There are nine destinations defined. These are locations of directories on the system.  Here is an example of a messages destination (ie. where syslog would normally send system messages to /var/log/messages)

destination d_mesg { file(“/var/log/messages”); };


There are eight filters declared.  This tells syslog-ng how at what level to filter the messages similar to /etc/syslog.conf for syslogd. Here is an example for the messages logs.

filter f_default { level(info..emerg and not (facility(mail) or facility(authpriv) or facility(cron)); };

This is defining the same thing for syslog-ng that the following defines for syslogd:



The log section puts the other sections together.  There are eight defined log entries.  The example for the /var/log/messages is below.

log { source(s_sys); filter(f_default); destination(d_mesg); };

This says any messages coming from source s_sys that match the filter f_default, send them to d_mesg location.

Stop syslog and start syslog-ng

  • Stop syslog service: service syslog stop
  • Disable it from starting on reboot: chkconfig syslog off
  • Start syslog-ng service: service syslog-ng start
  • Enable syslog-ng to start on reboot: chkconfig syslog-ng on

Configure Central Log Server

In general, configuration of Syslog-NG is well covered by Balazs Scheidler’s reference manual[1] and Nate Campi’s excellent FAQ[2]. So allow me to just present complete configuration examples for the main loghost and remote log server and point out the critical bits.

First let’s take a look at the configuration for the main loghost:

options {

use_fqdn (yes);






source s_inputs {

file (“/proc/kmsg” log_prefix(“kernel: “));

unix-stream (“/dev/log”);



tcp(ip(“”) port(514) max-connections(300));


#Destinations: local files, the console and the client files
destination r_logs { file (“/var/log/clients/$HOST/$YEAR/$MONTH/$FACILITY.$YEAR$MONTH$DAY”); };
# Log statements added for remote hosts
log { source (s_inputs); destination (r_logs); };

As far as the options go, “check_hostname(yes)” forces Syslog-NG to do a little bit of sanity checking on the incoming remote hostname in the log message. In our destination directive we’ll be creating directories for each system’s logs by hostname and it wouldn’t be good if an attacker could embed shell meta-characters in the hostname to cause us problems. “keep_hostname(yes)” means to use the hostname that’s presented in the actual message from the remote log server rather than using the hostname we get by resolving the source of the remote Syslog connection. After all, since we expect remote messages to be coming down our SSH tunnel, the source IP address of these messages will be the loopback address (, and having all messages tagged with “localhost” is not what we want.

The inputs cover all of the various places we can get logging information from. “internal()” is internal messages from the Syslog-NG daemon itself. “unix-stream(“/dev/log”)” is the normal /dev/log device that Linux systems use for local logging. Note that if you’re on a non-Linux platform like Solaris, HP-UX, or one of the *BSD operating systems then your local log channel is likely to be very different (examples of appropriate configurations for various operating systems can be found in the Syslog-NG source distribution). Some sites actually run the vendor Syslog in parallel with Syslog-NG rather than having to deal with the problem of emulating the standard vendor Syslog interfaces– the vendor Syslog daemon can just relay messages to Syslog-NG via the standard UDP Syslog channel, even within the same machine.

The “udp()” line means to listen on the standard 514/udp Syslog channel and “tcp()” means to listen on 514/tcp for messages from another Syslog-NG server (or in our case, the SSH tunnel). Note that both the “tcp()” and “udp()” options accept the “port()” option to specify a different port.

For example, if you wanted your Syslog-NG server to listen on port 5014/tcp to avoid conflicts with the rlogin/rsh daemon you would write:

tcp(port(5014) max-connections(100));

Note also the use of the “max_connections()” option to increase the number of simultaneous TCP sessions the logging daemon can handle.

The destination clause allows us to specify a “log sink”, or place where we want our logs to end up. Here we’re using some built-in Syslog-NG macros to force incoming log messages to be divided out into directories: first by hostname, and then by year and month. Within each directory, messages will go into log files named for the Syslog facility the message was logged to (mailauthkernlocal0, etc), with each file having a date stamp attached. Notice that with Syslog-NG automatically creating a new file for each day of logs, we don’t even need a separate log rotation program! This is just one more useful feature of Syslog-NG. The other options to the “file()” directive make sure that directories will be created as needed and set sensible ownerships and permissions on the newly created files and directories.

Having defined our inputs and destination directives, we combine them into log declarations to actually tell the Syslog-NG daemon what to do with the incoming messages. Here we’re just doing the trivial rule that sends all of our incoming messages from all sources into the log file directory hierarchy we defined in the destination directive above.

Configuring the Log Client

Here is the only thing I added on my client syslog-ng.conf file.

destination remote {
tcp(“” port(514));
log {

Restarted the syslog-ng service on the remote log client and logs appeared on the Central log server.