Open Source Software Technical Articles

Want the Best of the Wazi Blogs Delivered Directly to your Inbox?

Subscribe to Wazi by Email

Your email:

Connect with Us!

Current Articles | RSS Feed RSS Feed

Linux auditing 101


Anyone who is concerned about their Linux servers' security, stability, and proper functioning needs to audit their systems. Auditing may include anything from logging simple Bash commands to following complex system processes. Linux auditing can be a complex process, but here are some basic tools and techniques you can use to make the task simpler.

First, let's talk about some simple ways you can get some idea of what users do that don't reach the level of auditing. You can, for instance, check the shell command history using Bash's built-in command history. It shows the last commands executed by the current user. To see the date and time of executions, reconfigure history's settings with the command export HISTTIMEFORMAT='%F %T '.

The history command acts only upon the current user. To see other users' activity, provided you have permission, read the hidden file called .bash_history in their home directories. One caution, however: The Bash history can incomplete, making the record meaningless. If you see in it ./, all you know is that the user executed a script; you can't tell what was in this script or what it did, unless of course the script remains available and unchanged.

To see is who is currently logged in and what each user is doing, use the command /usr/bin/w, which gives you this information in full.

Many services, including a couple of popular databases, provide logs for simple auditing. For MySQL a hidden file called .mysql_history in users' home directories logs all the user actions in the MySQL console. A similar file for PostgreSQL is called .psql_history.

History logs frequently contain passwords or other sensitive data. Consider, for example, the commands they might show when you create MySQL or PostgreSQL users. Therefore, make sure that history log files can be read (and of course written) only by each user; they should have chmod permission flags set to 600. If you are more concerned about security than auditing, you can delete these history files entirely, then create a soft link to /dev/null in their place. For example, for MySQL, run the command ln -s /dev/null ~/.mysql_history.

These basic practices are just a start. They touch only the surface of what users do and what happens on the system, and it's trivial for an attacker to delete logs showing his traces. You therefore need an advanced auditing solution to not only reveal malicious activity but also store this information in a secure, preferably remote place.

Advanced Linux auditing

The Linux Auditing System is a Linux kernel implementation available in CentOS and other distributions that enables in-depth and advanced auditing. It works on the kernel level, where it can oversee every process and activity on the system. It uses the auditd daemon to log what it finds.

In most Linux distributions auditd is preinstalled and starts and stops automatically with the system. It logs information according to its auditing rules, and also conveys SELinux messages as described in our article about Linux server hardening.

Auditd's configuration is controlled by a few files in different directories. The daemon's configuration file is /etc/audit/auditd.conf, which contains all the settings except the auditing rules. Leave the default values in place while you're still exploring auditing. Important settings include max_log_file (default 6), the maximum size of the log file in megabytes. Once a log file reaches this limit, the action specified in the setting max_log_file_action (default rotate) takes place. The setting num_logs (default 5) specifies the number of log files and thus determines how many log files are kept.

The file /etc/audit/audit.rules contains the auditing rules that control what events should be audited and logged. You can specify three types of options in this file: control, file system, and system call.

The control options manage the system rather than the auditing rules. For example, the audit.rules file should always start with a directive that any existing auditd rules are deleted (-D). Another useful control option is -e 2, which makes the configuration immutable and requires a server restart for new changes to take effect.

File system rules pertain to files and directories recursively, and each rule looks like this:

-w path-to-file -p permission -k keyword

All file system rules begin with -w, which stands for watch. A permission is an action that reads (r), writes (w), executes (x), and/or changes the attribute (a) of a file. The keyword is an intuitive you choose to connect to one or more auditd rules. The same keyword can be used for more than one rule.

An example should help illustrate how file system rules work. The rule below instructs auditd to watch the file /etc/shadow, the Linux password file, for being read, written to or having its attributes modified:

-w /etc/shadow -p rwa -k shadow_watch

When a rule is tripped, auditd writes a log entry to its log file /var/log/audit/audit.log.

If you make any changes to the audit.rules file you must restart (or reload) the auditd daemon with the command service auditd restart. If you enter the rule above, you can test it by restarting the daemon, then trying to read /etc/shadow. Next, search the current autitd log using the ausearch command with the keyword for the rule in question: ausearch -k shadow_watch -i. The result should be similar to:

type=PATH msg=audit(11/18/2012 16:24:19.963:61) : item=0 name=/etc/shadow inode=163882 dev=fd:00 mode=file,000 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:shadow_t:s0 
type=CWD msg=audit(11/18/2012 16:24:19.963:61) :  cwd=/root 
type=SYSCALL msg=audit(11/18/2012 16:24:19.963:61) : arch=i386 syscall=open success=no exit=-13(Permission denied) a0=bfde58e9 a1=8000 a2=0 a3=1 items=1 ppid=2148 pid=2149 auid=root uid=anatoli gid=anatoli euid=anatoli suid=anatoli fsuid=anatoli egid=anatoli sgid=anatoli fsgid=anatoli tty=pts0 ses=1 comm=cat exe=/bin/cat subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=shadow_watch

The ausearch argument -i tells the command to interpret numbers; that is, uid 503, for instance, is translated to "anatoli." The above log excerpt shows that the user anatoli has tried to read the file /etc/shadow (name) using the executable /bin/cat (exe). The log shows success=no, the attempt was unsuccessful, and the exit code was -13, which means permission denied.

The ausearch utility lets you filter your results. To see all unsuccessful attempts, use -sv no, where sv stands for success value. The full command with the shadow keyword would be ausearch -k shadow_watch -sv no. For more information about the returned values and ausearch, check its manual page by running the command man ausearch.

The third type of option, system call (syscall), provides the interface between an application and the Linux kernel. These auditd rules act upon the specified interfaces to detect and log events. System call auditd rules have the following structure:

-a when,filter -S system-call -F field=value -k keyword

The first -a stands for append – that is, append the rule at the end of the ruleset. You can also use -A to place it at top of the list, or -d to delete the rule.

Possible when values are always and never, meaning always or never to create an event log.

For the third argument, filter, two values are frequently used: exit and user. The exit value means to act upon a syscall exit, when an operation is completed. The user filter is for userspace events and can be further filtered to uid, auid, gid, and pid.

The next argument is -S followed by a syscall name. There are hundreds of syscalls – to see all of them, go to the syscalls man page – and often more than one can be used to get a similar result. For example, if you want to see whether a file or directory has been deleted, you could use unlink, unlinkat, rename, or renameat.

The last argument before -k for keyword is -F, which stands for fine-tune filter field. If you go back to the ausearch -k shadow_watch result, you can see the numerous fields that can be used for fine-tuning the rules and hence the results. A sample fine-tuned rule looks like this:

-a always,exit -S unlink -S unlinkat -S rename -S renameat -F auid>=500 -F auid!=4294967295 -k delete

This rule covers syscalls that may lead to a file disappearing. The two fine-tuning rules (-F) state that in order the rule to be tripped the user's ID should be above 500 (regular users) but should be different from 4294967295, which is the representation of the auditd system ID. For further information about the fine-tuning fields, check the audit.rules man page.

To get you started faster with the auditd rules, CentOS 6 provides some example rules in the file /usr/share/doc/audit-2.2/stig.rules. Just using them without any adjustment provides a solid ground for auditing.

Auditd log interpretation

Once you've set the events you want to track, you can do the actual auditing by interpreting auditd's log file at /var/log/audit/audit.log. The file contains the same information given by ausearch but in more user-unfriendly format. One problem that you may encounter, if you decide to read the log file directly, is that the time is given in Unix timestamp format, which means you have to convert the timestamps to readable dates and times in order to tell when an event occurred.

Many tools can make it easier to read and analyze information from auditd's log. The aureport utility, for instance, lets you generate reports from the auditd log file. Running just aureport provides you with an easy-to-understand summary report. It includes counters for all important auditing event groups, such as number of changes to accounts, groups, or role. Detailed reports are also available, and they can be filtered by type (file system or syscalls), fields, and time. Here are a few useful aureport options:

  • --auth – shows authorization attempts. The aureport command can be further extended with the --failed argument to show failed attempts only and with --start to limit the time frame of the report. Thus to see failed logins for yesterday use the command aureport --auth --failed --start yesterday
  • --key – lists events for keywords defined in the auditd rules
  • --file – shows events for specified files and directories
  • --syscall – reports on system call events that have been configured to be logged in the auditd rules

For more details about these reports and to find out more information about aureport check its manual page (man aureport).

Auditing hardening

Auditd reporting has to provide genuine and reliable information. To make this possible the auditing cycle has to be secured and hardened. Auditing hardening enforces best practices for ensuring reliability, integrity, and security of the auditing process.

The first step for hardening is to ensure auditd's configuration is immutable by using the control option -e 2. Next, ensure the logs are stored in a secure centralized location. The best place is a server dedicated to accepting remote syslog events.

A utility called audispd (auditd's dispatcher) can help with this task, along with one of its plugins, audisp-remote. Audisp-remote allows events to be sent to a remote syslog server. Its configuration can be found in the /etc/audisp/audisp-remote.conf file. Here's a sample configuration specifying that a remote rsyslog server is listening on port TCP 514:

remote_server =
port = 514
transport = tcp

Having events logged and stored safely to a remote location helps provide peace of mind. Of course there's no guarantee that the remote server cannot be corrupted or compromised, but doing this adds one step more toward better system administration and security. That's why reliable remote logging is a requirement for financial and government environments.

Linux auditing can be as simple as reading simple history log files, or it can be a real challenge if you decide to be serious about it. For reliable auditing, use the powerful Linux kernel options and the supplementary services auditd and audispd.

This work is licensed under a Creative Commons Attribution 3.0 Unported License
Creative Commons License.


Very helpful explanation of flags and fields in audit.rules.
Posted @ Wednesday, August 28, 2013 4:56 PM by Lisa
>> The Linux Auditing System is a Linux kernel implementation available in CentOS  
auditd project is maintained by RedHat.
Posted @ Saturday, October 19, 2013 5:26 PM by lifriend
Do you have any good walk-thru's or white papers on configuring audispd/audisp-remote to send messaging to syslog on another server? 
The man pages don't cut it because there is no - "START HERE!" to figure out the starting point. 
I already have a centralized syslog configuration I documented and put into place, but I need this centralized auditing to work.
Posted @ Monday, April 28, 2014 7:55 AM by Warron
What you expect in a Handbag is everything you will get with ours.Our Chanel replica UK are of highest quality to feel and resemble the Original.You won't really notice any difference between the Original and our Chanel replica handbags.Why spend $700 or even thousands on an original?The quality of our Replicas are exactly the same.The difference, Price is the only difference.Don't pay a fortune for what you can get for below half of the Originals price. 
In this day and age with the economy hurting.It's good to know that you can get this quality for the money.Don't put yourself into a recession to get a quality Chanel replica purse.Get the same quality for an awesome price.You can look cool with your new bag and feel good that you didn't break the bank.Now you have some extra money to treat yourself to the finer things.Maybe you want to be able to buy a new dress to go with your new Handbag.Well with the money your saving you could afford to buy a few dress's.Keep some money to do some other things you wanted.Treat yourself to an evening out.You'll be able to with the money you save with our replica Chanel COCO.You won't even have to pay for shipping.Shipping is all included in our prices.So you can sit back with a piece of mind.You saved big bucks and treated yourself as well.Or you could pay the full price of an Original bag.Then you wouldn't have the extra money.Replica Handbags are truly the way for you to be in style and save money.Don't just take it from shop us for your Replica Handbags today.You will be glad you did, Your purse will be glad you did as well.
Posted @ Wednesday, July 23, 2014 4:28 AM by lready
Socialite Olivia Palermo exposed last month in louis vuitton replica cooperation with the Italian footwear brand Aquazzura, and launched Aquazzura × Olivia Palermo series, OP still own official blog in the first exposure of the three US-shoe manuscript. Poster hublot replica compiled code has also been concerned about the design of the finished product the ladies footwear, although we had to wait until September to buy shoes in the series, but Aquazzura official website chanel replica or released several Aquazzura × Olivia Palermo series of physical map, you can social queen's taste cartier replica really seen not let us down, kind of picture than the draft design more beautiful.  
Posted @ Friday, August 22, 2014 4:49 AM by VV
Post Comment
Website (optional)

Allowed tags: <a> link, <b> bold, <i> italics