Samhain Labs

By Rainer Wichmann rainer@nullla-samhna.de    (last update: Dec 22, 2010)

One of the most frequent questions that arise in connection with host integrity monitoring is: If a file has been modified, then who did it?

The problem with this question is that contrary to e.g. ownership or modification time, the UID of the user performing a modification is not recorded in the metadata of a file. And as we will see immediately, even if this would be done, it wouldn't really solve the problem.

The aim of this article is to show you how you can obtain the desired information by using the Linux Audit System, which is a powerful tool to obtain information about events happening on a Linux system.

Who is this user, really?

Suppose for a moment that an ls -l filename would tell you that the file has last been edited by the user rainer, what does it tell you? On second thought, not very much.. e.g. because, given appropriate privileges, users can change their identity with (sudo) su new_name. It might have been a system administrator. Or some other user with sufficient privileges. Or the cron daemon. And things will even get worse if ls -l filename tells you that it was root who did it.. who is root? Cron? The udevd deamon (just for example)? An actual person? Which one of several administrators?

The Audit User IDentity - AUID

Clearly, we first need a way to track a user, be it an actual person or a system user, in such a way that we won't lose track e.g. by a "sudo" or "su". The way this is done is by setting an additional UID, the AUID, which is different from the "normal" UID a user has, and is supposed to remain unchanged, whatever the user does.

This is done using PAM (Pluggable Authentication Modules), which is the standard way on Linux to handle authentication tasks. If you take a look into the directory /etc/pam.d, you will find there a number of files named after services that use PAM for authentication, such as e.g. /etc/pam.d/gdm, /etc/pam.d/login, and /etc/pam.d/sshd. You might also find there files like /etc/pam.d/common-session or /etc/pam.d/common-auth, which contain common directives and are included by multiple other files.

What needs to be done now is to add the following line:

session required pam_loginuid.so

for every login service or system user that we are interested in, like: /etc/pam.d/atd, /etc/pam.d/cron, /etc/pam.d/gdm, /etc/pam.d/login, and /etc/pam.d/sshd.

Do not include this line in files like /etc/pam.d/su or /etc/pam.d/sudo. This line will call the pam_loginuid module to set the AUID on login, and we do not wish to reset the AUID with a "sudo" or "su", because this would lose track of the login user.

Getting at the information - The Audit Deamon

Installation

On Debian or Ubuntu, the Linux audit daemon is in the auditd package.

sh# apt-get install auditd
sh# service auditd status
 * auditd is running.

On Redhat or CentOS, the package name is audit (but the service name is still auditd).

sh# yum install audit
sh# service auditd status
auditd (pid nnn) is running...

Audit rules don't automatically survive a reboot! You may want to save your rules in a file and execute this file upon reboot (e.g. you could add your rules to /etc/rc.local).

You need to tell the audit deamon for which file(s) and/or directories access should be monitored. E.g. if you want to monitor changes to /etc/group, the command would be

sh# auditctl -w /etc/group -p wa -k mykey

Here, "-p wa" tells the audit deamon that you want to watch write (w) and attribute change (a). There are two more options available: read (r) and execute (x).

The "-k mykey" specifies an arbitrary string "mykey" that you can use to search audit reports related to this particular rule (or rules, if you use the same key for multiple rules).

What can you watch?

You can place watches on arbitrary files or directories, but:

• you cannot place a watch on the root directory, and
• you cannot use wildcards.

Watches placed on a directory are recursive, i.e. they include all subdirectories.

How can I delete a rule to watch a file?

sh# auditctl -D -k mykey

will delete all rules with the key "mykey".

sh# auditctl -W /etc/group

will remove a watch for the path "/etc/group" (lowercase '-w' to set a watch, uppercase '-W' to remove a watch).

To get the audit report on changes, you can use:

sh# ausearch -i -k mykey

which will give you the reports on every rule associated with the key "mykey". As you can see in the example output below, you will see multiple lines of detail pertaining to a change (in this case, user "root" has edited the file /etc/group using the "vi" editor).

First, in this particular example you will notice that there are three groups of reports, ordered by time. Each group contains a single type=SYSCALL item, which describes what basic system call has been executed, and several other items describing pertinent details. So, if you carefully look at the following three group of lines, you will notice that:

• they always refer to the path name=/etc/group,
• they all refer to the executable exe=/usr/bin/vim.basic, which is the program used by the person editing that file,
• they all point out that the user editing the file was uid=root, and had logged in as auid=rainer, and
• they describe a sequence of three events happening to the edited path name=/etc/group, ordered by time:
1. at 11:36:37.357:23, the path was renamed (syscall=rename) to name=/etc/group~
2. at 11:36:37.357:25, it was opened (syscall=open), presumably to write the new version, and finally
3. at 11:36:37.393:26, the file permissions were changed (syscall=chmod), most likely to reset them to the previous value.

Record group 1: rename /etc/group to /etc/group~

type=PATH msg=audit(12/22/2010 11:36:37.357:23) : item=3 name=/etc/group~ 
  inode=278539 dev=08:06 mode=file,644 ouid=root ogid=root rdev=00:00 
type=PATH msg=audit(12/22/2010 11:36:37.357:23) : item=2 name=/etc/group 
  inode=278539 dev=08:06 mode=file,644 ouid=root ogid=root rdev=00:00 
type=PATH msg=audit(12/22/2010 11:36:37.357:23) : item=1 name=/etc/ inode=260611 
  dev=08:06 mode=dir,755 ouid=root ogid=root rdev=00:00 
type=PATH msg=audit(12/22/2010 11:36:37.357:23) : item=0 name=/etc/ inode=260611 
  dev=08:06 mode=dir,755 ouid=root ogid=root rdev=00:00 
type=CWD msg=audit(12/22/2010 11:36:37.357:23) :  cwd=/root 
type=SYSCALL msg=audit(12/22/2010 11:36:37.357:23) : arch=i386 syscall=rename 
  success=yes exit=0 a0=9164890 a1=91cf6f8 a2=9164890 a3=91cf6f8 items=4 ppid=2026 
  pid=2168 auid=rainer uid=root gid=root euid=root suid=root fsuid=root egid=root 
  sgid=root fsgid=root tty=pts0 ses=1 comm=vi exe=/usr/bin/vim.basic key=mykey 

Record group 2: open /etc/group

type=PATH msg=audit(12/22/2010 11:36:37.357:25) : item=1 name=/etc/group 
  inode=265901 dev=08:06 mode=file,644 ouid=root ogid=root rdev=00:00 
type=PATH msg=audit(12/22/2010 11:36:37.357:25) : item=0 name=/etc/ inode=260611 
  dev=08:06 mode=dir,755 ouid=root ogid=root rdev=00:00 
type=CWD msg=audit(12/22/2010 11:36:37.357:25) :  cwd=/root 
type=SYSCALL msg=audit(12/22/2010 11:36:37.357:25) : arch=i386 syscall=open 
  success=yes exit=3 a0=9164890 a1=241 a2=1a4 a3=0 items=2 ppid=2026 pid=2168 
  auid=rainer uid=root gid=root euid=root suid=root fsuid=root egid=root 
  sgid=root fsgid=root tty=pts0 ses=1 comm=vi exe=/usr/bin/vim.basic key=mykey 

Record group 3: change permissions on /etc/group

type=PATH msg=audit(12/22/2010 11:36:37.393:26) : item=0 name=/etc/group 
  inode=265901 dev=08:06 mode=file,644 ouid=root ogid=root rdev=00:00 
type=CWD msg=audit(12/22/2010 11:36:37.393:26) :  cwd=/root 
type=SYSCALL msg=audit(12/22/2010 11:36:37.393:26) : arch=i386 syscall=chmod 
  success=yes exit=0 a0=9164890 a1=81a4 a2=1 a3=0 items=1 ppid=2026 pid=2168 
  auid=rainer uid=root gid=root euid=root suid=root fsuid=root egid=root 
  sgid=root fsgid=root tty=pts0 ses=1 comm=vi exe=/usr/bin/vim.basic key=mykey 

One more complication - Booting it up

The audit daemon enables the audit system in the kernel. Therefore all processes starting earlier than the audit daemon itself may get an auid=4294967295 (unknown). To avoid this problem, you may want to add audit=1 to the kernel boot parameters.