Samhain Labs

By Rainer Wichmann rainer@nullla-samhna.de    (last update: Dec 29, 2009)

Introduction

Scenarios for break-ins / intrusions commonly assume that the intruder will gain superuser (root) privileges, and will manage to completely subvert the host after an intrusion.

However, usually the intruder does not have root privileges initially, either because:

• the break-in was performed by exploiting a vulnerability in some network deamon running as a non-root user, or
• the break-in happened by social engeneering or by (brute-force) guessing the password of a non-root user.

• your machines are well patched, and/or
• the intruder (only) uses some outdated exploits that are ineffective against up-to-date patched hosts.

While the last point sounds a bit optimistic, experience shows that a sizeable fraction of intruders are not researchers equipped with bleeding-edge tools, but rather people with somewhat limited knowledge, relying on a small set of programs/exploits that are known to them, but may not work on well patched hosts.

What will the intruder do?

Even without superuser privileges, the machine is quite useful for an intruder, e.g. to:

• perform scanning in order to locate other vulnerable hosts on the network,
• install a file sharing server (serving from a nonstandard, non-privileged port) to distribute illegal content (e.g. pirated software or movies),
• use the host as a spam relay, or
• establish a web server (again on a nonstandard, non-privileged port) to be used for some phishing scam.

How do you determine the presence of a nonprivileged intruder?

Having gained superuser privileges, intruders often start to fuss about the machine in order to install some rootkit, with the aim to hide their presence and secure their access. This, however, modifies the operating system, and thus actually helps to detect them. Sometimes the activity of an intruder even may botch up the operating system (e.g. by a buggy rootkit), leading to prompt detection of the break-in.

On the other hand, legitimate nonprivileged shell users notoriously tend to do lots of random and unpredictable things, often with no useful pattern. Having users that travel around the world and ssh into your hosts at random times doesn't help either. I.e. there is a lot of 'noise' that helps an intruder to 'blend in', and can make it difficult to detect an intrusion.

Things to watch out for include (but may not be limited to):

Suspicious outbound traffic

Usually, an intruder will try to use your machine(s) in order to gain access to other machines, e.g. by running portscans and/or network attacks. Unless the intruder is cautious and performs these activities very slowly, this will generate very unusual and suspicious outbound network traffic that can be captured with a network sniffer such as e.g. snort.

Suspicious open ports

Intruders may use your hosts as a depot for illegal content like e.g. "pirated" copies of movies. Distributing such content requires some network application (e.g. an ftp server or a peer-to-peer application) that opens a port to listen for incoming connection attempts.

Information on open TCP ports may be collected with the command netstat -pant | grep LISTEN or alternatively lsof -i tcp | grep LISTEN

Likewise, information on open UPD ports can be collected with the command netstat -panu | grep -v ESTABLISHED

Unfortunately, automated evaluation of such information may get complicated by user applications which open ports for more or less legitimate reasons. This doesn't only apply to notorious applications like skype - there are quite a few ill-coded application suites that rely on sockets for interprocess communication, and by default open a TCP socket for all interfaces, even if in most cases a Unix Domain Socket would suffice. For this reason, you may want to monitor open ports in order to learn how the system looks normally.

Suspicious established connections

There are file sharing programs which can log in to an IRC channel, wait for commands, and then initiate a connection to a designated IP address in order to deliver content (let's say, pirated movies or software).

As you may notice, this scheme completely avoids any open ports. All connections are outbound connections from the hijacked machine. Checking for open ports is useless if an intruder has installed such software.

For this reason, it is highly recommended to run regular checks for established connections, and familiarize yourself with the 'normal' pattern of software and/or outbound connections of legitimate users, such that you are able to spot anomalies, like established connections involving new and/or unusual software. To check for established TCP connections, you can use: netstat -pant | grep ESTABLISHED

Disk full

Some intruders manage to botch the system by exhausting the diskspace available for /tmp or /var/tmp. This may lead to obvious problems if other applications cease to function properly.

Suspicious file names

You may want to look out for files or directories like ".. " (note trailing whitespace) or " " in /tmp and in user home directories. With GNU ls, using ls -Q is helpful (this encloses filenames in double quotes).

Crontab, ~/.profile, ~/.bashrc

If an intruder runs some service from your host(s), then he/she is probably interested in the service surviving a reboot. Since an unprivileged user cannot start services during the boot sequence, the intruder has to resort to other methods. One possibility would be to install a cron job that checks at regular intervals whether the service is running, and restarts it if required.

Another option would be to modify the ~/.profile or ~/.bashrc file of the user (if the intruder has gained access to a shell account). In this case, whenever the legitimate user starts a shell, the code implanted by the intruder would run.

Recovering the actions of the intruder

Finding out what an intruder has done is important for assessing the damage, and cleaning up after detection of the intrusion. The following is a (probably incomplete) list of things that will help you in this task.

The shell history

I will only discuss the bash shell here, but most Bourne-style shells should be similar. Usually, the intruder will try to hide his actions by preventing the shell from writing the history file. This can be done by any of the following means:

• Remove the history file ($HOME/.bash_history), then create a directory with the same name (or a link to /dev/null).
• Unset the HISTFILE shell environment variable, or set it to /dev/null.
• Set the HISTSIZE and/or HISTFILESIZE shell environment variable(s) to 0 (zero).

It is not possible to fully prevent a user from evading the shell command history. However, the following steps will serve to protect against the most commonly used methods (i.e. those shown above):

1. Use file flags to make the history file of your users append-only.
   • On Linux, set the 'append-only' attribute on the history file with
     chattr +a ~username/.bash_history
     (may not be supported by all filesystem types). See 'man chattr/lsattr' for details.
     Not feasible for NFS-mounted home directories (no protocol support). Read/write operations would be impossible.
   • On FreeBSD, set the 'append-only' file flag with
     chflags sappnd ~username/.bash_history
     See 'man chflags' for details. Probably will also not work for NFS-mounted home directories.
   These restrictions also affect the superuser (root), i.e. the file flag must be removed before the file can be truncated/deleted. On FreeBSD, running in securelevel 1 or higher will prevent any user from removing the sappnd flag.
2. Use the readonly shell builtin to make the relevant shell environment variables readonly. E.g. you could include the following lines in /etc/profile:

   readonly HISTFILE
   readonly HISTFILESIZE
   readonly HISTIGNORE
   readonly HISTSIZE

Process accounting

Process accounting allows an administrator to track resource usage by users as well as the commands executed by them. The primary advantage is that process accounting cannot be disabled by unprivileged users (even more, they can't even determine whether it takes place).

There are however also significant disadvantages:

• It may generate large logfiles, requiring a lot of diskspace,
• only the command itself is recorded, not the arguments (thus e.g. you can learn that a user has run rm, but you won't know which file has been removed), and
• process accounting information gets written only when a process exits, i.e. as long as a process still runs, it will not show up.

On Debian/Ubuntu Linux, 'apt-get acct' will install the userspace accounting utilities and enable accounting. On Fedora/RHEL, the package is named 'psacct', and after installation, you need to start process accounting manually with:

# chkconfig psacct on
# /etc/init.d/psacct start

For FreeBSD, to enable process accounting use:

# touch /var/account/acct
# accton /var/account/acct
# echo 'accounting_enable="YES"' >> /etc/rc.conf

After running your system for some time (to let a bit of logging information accumulate), you can then use the 'lastcomm' command to get information about user activity. E.g.: lastcomm rainer may yield information like:

bash              F    rainer   stderr     0.00 secs Wed Dec 27 12:12
sed                    rainer   stderr     0.01 secs Wed Dec 27 12:12
bash              F    rainer   stderr     0.00 secs Wed Dec 27 12:12
bash              F    rainer   stderr     0.00 secs Wed Dec 27 12:12
tr                     rainer   stderr     0.01 secs Wed Dec 27 12:12
wc                     rainer   stderr     0.00 secs Wed Dec 27 12:12
...