Stop net-snmp log messages

I am setting up a central logging host and have a central monitoring host that monitors all my servers with snmp queries.  This was filling up my log files so in order to disable this logging:

* I edited the /etc/init.d/snmpd file and removed the -Lsd option at the front of the OPTIONS variable.

* Restarted snmpd daemon: service snmpd restart

Secure central log server with syslog-ng and SSH

SSL vs. SSH

I chose to port forward syslog-ng traffic over SSH versus configuring syslog-ng over SSL due to the fact that I already had a port open on my servers for SSH and due to the fact that SSL attempts to make the connection with unencrypted channels and SSH encrypts all communications to and from the client and server. Here is an excellent how to for tunneling over SSH: http://www.deer-run.com/~hal/sysadmin/SSH-SyslogNG.html another helpful link was this: http://www.mernin.com/blog/?p=83

Setup the tunnel

The first step is to get the SSH tunnel set up between the two machines. My personal preference is to originate the SSH tunnel on my central “loghost” machine at the primary site, and have it connect to the machine at the “remote” site that I want to get logs from. Typically this involves drilling out through the firewall at the primary site– often the site’s default firewall rules will allow this connection without any reconfiguration– and allowing the connection “inward” through the firewall at the remote end, which usually requires some firewall ruleset tweaks on the remote site’s firewall.

However, since we want the remote log server to be sending logs back to the central loghost at the primary site, we need to use a reverse tunnel (that’s the “-R” option on the SSH command line) to get things working properly. This is actually one of very few places where I find reverse tunnels to be useful. Figure A, below, shows a high-level picture of how the traffic is flowing in this design.

We need to make sure that the SSH session and tunnel are set up automatically when the central log host boots. If the SSH session dies for some reason (intermittent network outage, system administration “accident”, etc) we’d also like the connection to be re-established as quickly as possible. In situations like this, I like to have the init process fire off the SSH connection with a line like this in /etc/inittab:

log1:3:respawn:/usr/bin/ssh -nNTx
            -R 514:loghost.domain.com:514
            remote.domain.com >/dev/null 2>&1

The example above must appear as a single long line in /etc/inittab— I’ve just broken it onto multiple lines for clarity.

Let’s examine the SSH command line first. The “-R 514:loghost.domain.com:514” on the second line of the example sets up the reverse tunnel from 514/tcp on the remote server to “loghost.domain.com:514“– in other words, port 514/tcp on the central loghost machine. While it seems natural to use 514/tcp for Syslog-NG logging, you have to remember that 514/tcp is the reserved port for the Unix rlogin/rsh service so you’re going to run into a port conflict if you still have these services enabled. I generally turn off unencrypted network protocols like telnet, FTP, rlogin/rsh/rcp on my servers and use SSH instead, so it’s not an issue for me, but you can run this tunnel over any free ports if there is a conflict at your site.

As for the other SSH command line options above, the “-n” flag tells SSH to associate the standard input with /dev/null. There won’t be any command line input since we’re essentially going to be running the SSH client as a “daemon” via init. As you can see at the end of the command line in the example, we’re also sending the standard output and standard error to /dev/null as well (“… >/dev/null 2>&1“). Since we’re never going to be issuing remote commands via this SSH connection (we only care about the tunnel), the “-N” option to SSH tells the SSH client to only set up the tunnel and to not bother preparing a command stream for issuing commands on the remote system, while “-T” says to not bother allocating a pseudo-tty on the remote system. The “-x” option disables X11 forwarding, just as a defense-in-depth gesture.

Turning our attention to the rest of the /etc/initab entry, the first field (“log1“) is just an identifier for this entry in the inittab file. These identifiers can be any sequence of 2-4 alphanumeric characters; the only requirement is that they be unique from all other identifiers used in the file. I’ve chosen “log1” here because it’s usually the case that I have multiple SSH tunnels set up to different remote log sources, and I typically name the inittab entries “log1“, “log2“, etc. The second field in the inittab file (“3“) is the run level where this entry should be fired. Make sure to start this SSH process after the network interfaces have been initialized but before the Syslog-NG daemon is started.

The “respawn” option in the third field is the reason I like to use init for spawning processes like this. When the “respawn” option is enabled, the init process will automatically fire off a new SSH process if the old one dies for any reason. In other words, init acts a like a “watchdog” type daemon and makes sure that the SSH tunnel is always up and running. This is an extremely useful technique, but one that a lot of system admins seem to have forgotten.

Once you’ve got your inittab entry all set up, HUP the init process (“kill -HUP 1“). This should cause the init process to re-read the inittab file and spawn the SSH connection. You should be able to verify that the SSH client is running with the ps command and verify the existence of the tunnel using netstat. Once you’ve got all that working, it’s time to turn our attention to configuring Syslog-NG.

Test the connection via command line and make sure you are not prompted for a password.

server # /usr/bin/ssh -nNTx -R 1514:127.0.0.1:514 someuser@remotehost.domain.com

Installing Syslog-NG

Download the appropriate .rpm from http://pbone.net for your system

Install with rpm -ivh syslog-ng.*.rpm

Default syslog-ng configuration

You can view the locations of all the files and binaries installed via the rpm package: rpm -ql syslog-ng

The good thing about and rpm package is that it installs init scripts and logrotate config files for you.

Sections

  • Options – used to configure global options
  • Source – One or more source sections are used to control where messages may originate
  • Destination – One or more destination sections declare where messages should be sent
  • Filter – Declare filters that should be applied to messages.
  • Log – Include one or more sources, one or more filters, and one or more destinations

The default syslog-ng.conf file that comes with RHEL5 is configured to perform like the syslogd service that is installed by default.

Source:

There is one source configured for the system called s_sys. It is defined here:

source s_sys {
file (“/proc/kmsg” log_prefix(“kernel: “));
unix-stream (“/dev/log”);
internal();
#udp(ip(0.0.0.0) port(514));
};
Destination:

There are nine destinations defined. These are locations of directories on the system.  Here is an example of a messages destination (ie. where syslog would normally send system messages to /var/log/messages)

destination d_mesg { file(“/var/log/messages”); };

Filters:

There are eight filters declared.  This tells syslog-ng how at what level to filter the messages similar to /etc/syslog.conf for syslogd. Here is an example for the messages logs.

filter f_default { level(info..emerg and not (facility(mail) or facility(authpriv) or facility(cron)); };

This is defining the same thing for syslog-ng that the following defines for syslogd:

*.info;mail.none;authpriv.non;cron.none

Log:

The log section puts the other sections together.  There are eight defined log entries.  The example for the /var/log/messages is below.

log { source(s_sys); filter(f_default); destination(d_mesg); };

This says any messages coming from source s_sys that match the filter f_default, send them to d_mesg location.

Stop syslog and start syslog-ng

  • Stop syslog service: service syslog stop
  • Disable it from starting on reboot: chkconfig syslog off
  • Start syslog-ng service: service syslog-ng start
  • Enable syslog-ng to start on reboot: chkconfig syslog-ng on

Configure Central Log Server

In general, configuration of Syslog-NG is well covered by Balazs Scheidler’s reference manual[1] and Nate Campi’s excellent FAQ[2]. So allow me to just present complete configuration examples for the main loghost and remote log server and point out the critical bits.

First let’s take a look at the configuration for the main loghost:

options {

use_fqdn (yes);

perm(640);

dir_perm(0750);

check_hostname(yes);

keep_hostname(yes);

};

source s_inputs {

file (“/proc/kmsg” log_prefix(“kernel: “));

unix-stream (“/dev/log”);

internal();

udp(ip(“127.0.0.1”));

tcp(ip(“127.0.0.1”) port(514) max-connections(300));

};

#Destinations: local files, the console and the client files
destination r_logs { file (“/var/log/clients/$HOST/$YEAR/$MONTH/$FACILITY.$YEAR$MONTH$DAY”); };
# Log statements added for remote hosts
log { source (s_inputs); destination (r_logs); };

As far as the options go, “check_hostname(yes)” forces Syslog-NG to do a little bit of sanity checking on the incoming remote hostname in the log message. In our destination directive we’ll be creating directories for each system’s logs by hostname and it wouldn’t be good if an attacker could embed shell meta-characters in the hostname to cause us problems. “keep_hostname(yes)” means to use the hostname that’s presented in the actual message from the remote log server rather than using the hostname we get by resolving the source of the remote Syslog connection. After all, since we expect remote messages to be coming down our SSH tunnel, the source IP address of these messages will be the loopback address (127.0.0.1), and having all messages tagged with “localhost” is not what we want.

The inputs cover all of the various places we can get logging information from. “internal()” is internal messages from the Syslog-NG daemon itself. “unix-stream(“/dev/log”)” is the normal /dev/log device that Linux systems use for local logging. Note that if you’re on a non-Linux platform like Solaris, HP-UX, or one of the *BSD operating systems then your local log channel is likely to be very different (examples of appropriate configurations for various operating systems can be found in the Syslog-NG source distribution). Some sites actually run the vendor Syslog in parallel with Syslog-NG rather than having to deal with the problem of emulating the standard vendor Syslog interfaces– the vendor Syslog daemon can just relay messages to Syslog-NG via the standard UDP Syslog channel, even within the same machine.

The “udp()” line means to listen on the standard 514/udp Syslog channel and “tcp()” means to listen on 514/tcp for messages from another Syslog-NG server (or in our case, the SSH tunnel). Note that both the “tcp()” and “udp()” options accept the “port()” option to specify a different port.

For example, if you wanted your Syslog-NG server to listen on port 5014/tcp to avoid conflicts with the rlogin/rsh daemon you would write:

tcp(port(5014) max-connections(100));

Note also the use of the “max_connections()” option to increase the number of simultaneous TCP sessions the logging daemon can handle.

The destination clause allows us to specify a “log sink”, or place where we want our logs to end up. Here we’re using some built-in Syslog-NG macros to force incoming log messages to be divided out into directories: first by hostname, and then by year and month. Within each directory, messages will go into log files named for the Syslog facility the message was logged to (mailauthkernlocal0, etc), with each file having a date stamp attached. Notice that with Syslog-NG automatically creating a new file for each day of logs, we don’t even need a separate log rotation program! This is just one more useful feature of Syslog-NG. The other options to the “file()” directive make sure that directories will be created as needed and set sensible ownerships and permissions on the newly created files and directories.

Having defined our inputs and destination directives, we combine them into log declarations to actually tell the Syslog-NG daemon what to do with the incoming messages. Here we’re just doing the trivial rule that sends all of our incoming messages from all sources into the log file directory hierarchy we defined in the destination directive above.

Configuring the Log Client

Here is the only thing I added on my client syslog-ng.conf file.

destination remote {
tcp(“127.0.0.1” port(514));
};
log {
source(s_inputs);
destination(remote);
};

Restarted the syslog-ng service on the remote log client and logs appeared on the Central log server.

Change timezone in Linux

  1. Check the current time. Login as root and execute the date command: # date
  2. Backup your current localtime file: cp /etc/localtime /etc/localtime.bak
  3. Create a symbolic link to the appropriate timezone from /etc/localtime: ln -sf /usr/share/zoneinfo/America/Los_Angeles /etc/localtime
  4. Set the ZONE entry in the file /etc/sysconfig/clock (ex. “America/Los_Angeles”)
  5. Set the hardware clock by executing: /sbin/hwclock –systohc
  6. Or if you have ntp installed you can set the clock with ntpdate command: ntpdate -b time.nist.gov

Resource Mate on Linux

Get setupunix.ext from Resource Mate

This is a Windows executable. It is purposely renamed to get through email.  Rename the file setupunix.exe and run it on a Windows system.   This will extract the necessary files you need for the installation

Copy the files to the web server

*Copy the cgi-bin folder to the DocumentRoot of the web server or VirtualNamedHost. This should be what you have once copied up to the server. /path/to/DocRoot/cgi-bin/tmp

* Make sure the permissions on cgi-bin and tmp directories are 755. In other words readable and executable by everyone.

* Change the libsearch.cgi file to be executable: chmod 755 cgi-bin/tmp/libsearch.cgi

* Edit the lib.conf file with the appropriate information.  A detailed explanation is enclosed as a word doc with the extracted files.

Add ScriptAlias to Apache httpd.conf

* Edit the httpd.conf file to add a script alias for the cgi-bin directory: ScriptAlias /cgi-bin/ “/path/to/DocRoot/cgi-bin/tmp”

* Reload Apache: sudo /sbin/service httpd reload

Browse to URI

* Go to the URI of the site to the libsearch.cgi executable.  http://www.site.com/cgi-bin/libsearch.cgi

Dependencies

* The libsearch.cgi is a perl script.  The following perl modules were required on my system to get the libsearch.cgi script to work

  • LWP::Simple
  • ExtUtils::Installed
  • CGI::Carp
  • CGI::Session
  • Cwd

OS X Leopard Rotate Logs and Restart Zope

I recently ran into an issue where our Zope process was crashing.  We have 4 Zope processes installed, 3 are load balanced with Zeo and the forth is for images files.

I created a script that would check the size of the log files for each instance and if the size of the log files was bigger than 1GB to rotate the logs and restart that particular Zope instance. The name of my script was logrotate.sh.

Now I wanted this script called at a specified day of the week and time so I couldn’t just add it to /etc/periodic/weekly, well I could and then change the time that the weekly scripts run but I didn’t want to change that system wide.  If I wanted to change it though you would edit /System/Library/LaunchDaemons/com.apple.periodic-weekly.plist and edit the Hour, Minute, and Weekday integers as appropriate.

So I created my own plist and added it to /System/Library/LaunchAgents/edu.ucla.zoperestart.plist

Here’s what my plist looks like:

<?xml version=”1.0″ encoding=”UTF-8″?>
<!DOCTYPE plist PUBLIC “-//Apple Computer//DTD PLIST 1.0//EN” “http://www.apple.com/DTDs/PropertyList-1.0dtd”&gt;
<plist version=”1.0″>
<dict>
<key>Debug</key>
<false/>
<key>Label</key>
<string>ZopeRestart</string>
<key>Nice</key>
<integer>10</integer>
<key>ProgramArguments</key>
<array>
<string>/path/to/scripts/logrotate.sh</string>
</array>
<key>UserName</key>
<string>root</string>
<key>StartCalendarInterval</key>
<dict>
<key>Hour</key>
<integer>9</integer>
<key>Minute</key>
<integer>15</integer>
<key>Weekday</key>
<integer>4</integer>
</dict>
</dict>
</plist>

Now I had to load the plist so that launchctl was aware of it and would read the xml file and run it at the appropriate time.

launchctl load /System/Library/LaunchAgents/edu.ucla.zoperestart.plist

Verify that the plist is loaded:

launchctl list

I purposely set the date and time to 15 minutes in the future just to test that launchd ran my script.  Now it’s working great!

Resolve OS X folder with question mark

Today I applied security patches to one of my Leopard servers and rebooted only to have the server not come up.  I had to drive all the way into work to discover the folder with a question mark on it signifying that the server didn’t recognize any boot drive.

Found this very helpful post: http://support.apple.com/kb/TS1440?viewlocale=en_US

Fortunately I could see the drive and the software RAID through Disk Utility when booting from CD.

Tried all of those steps and no luck!  I really didn’t want to reformat the drive and re-install the OS and then have to re-install all applications too!

Recognizing that I had a mirror RAID with 2 disks I thought maybe one of the disks might be causing and issue so I removed one of the disks from the RAID set like this.

dlopez$ diskutil checkRAID
RAID SETS
---------

Name:                   Mirror
Unique ID:              2CFEC74D-5C0C-49CB-940E-6099AE3C2B97
Type:                   Mirror
Status:                 Online
Device Node:            disk4
Apple RAID Version:     2
----------------------------------------------------------------------
 #      Device Node     UUID                                    Status
----------------------------------------------------------------------
0       disk1s3         EF3F0D87-0E9A-4ED4-BB2B-F2A7EBFD0306    Online
1       disk3s3         B1EC4F22-9C0D-4CDB-90FE-D49347195834    Online
----------------------------------------------------------------------

I wanted to remove disk3s3 from the RAID

dlopez$ sudo diskutil removeFromRAID disk3s3 disk4
Password:
AppleRAID Headers removed from disk 'disk3s3'
Changing filesystem size on disk 'disk3s3'...
Attempting to change filesystem size from 14658928640 to 14658936832 bytes
The disk has been removed from the RAID

Then I ran a disk repair on it and rebooted.  Sure enough the server was back!  Now I just need to recreate the mirror RAID again.