Nowadays many Linux servers are neither initially installed nor maintained by dedicated Linux administrators, yet IT professionals are responsible for their servers' security. If your server is compromised, not only can all your sensitive information be exposed, but your server can also become a tool in the wrong hands. To avoid such scenarios, make sure your server is properly configured and regularly updated, and perform frequent security audits using the tools and procedures below.
The most effective Linux security audit is specially tailored to the applications and services running on your server. This means that you have to know the environment you are auditing. This allows you to determine where potential security risks are hidden and where your security scan should begin. For example, running a web server introduces the risk of vulnerable web scripts, which are one of the most common targets for hacker attacks. Not to mention that the running web server itself, or any of its modules, can be outdated and vulnerable or insecurely configured.
A successful audit also requires some understanding of networking, programming (Perl, PHP, or other languages supported on your server), and Linux. It may require time or skills you don't have. However, sometimes you have to do it anyway. For instance, audits are essential when your datacenter gets complaints about suspicious activities from your server, such as spam or hacking attempts. It pays to have someone in your organization be acquainted with these techniques – or consider hiring a professional for this purpose. If you cannot afford security audits, consider hiring a managed Linux server provider or moving to a shared server where the provider takes all the responsibility for the security audits.
Once you are ready for the security audit follow these steps:
Penetration testing (pentesting) allows you to discover vulnerabilities in your server and evaluate its overall security. Such an evaluation is the basis of every security audit. It gives you applicable instructions on what and how to improve for the security of your server, and provides important information about where to concentrate your further efforts in your audit.
To perform a pentest you can use a vulnerability scanner such as Nessus, which has plugins for almost any kind of online service. While Nessus is the most popular and advanced vulnerability scanner, you can also try other tools such as Nmap, though it is primarily a port scanner and not exactly a vulnerabilty scanner; Metasploit, sophisticated but costly tool; or Backtrack Linux, a whole Linux distribution with a huge collection of pentesting tools. No matter which tool you choose or how you perform your pentesting you are certain to find some vulnerabilities, though they might be minor. This proves a popular theory that any resource or service exposed to the public should be considered potentially vulnerable and should be closely monitored. That's where the next phase of the security audit comes in: inspect logs and scan files.
PCI data security standards and assessments are prepared by a council representing the payment card industry. These frequently mandatory standards and procedures are created to ensure the safety of sensitive information such as credit card details stored in payment processors. Most of these standards can be adapted in other industries for properly segmenting networks, configuring sufficient logging, efficient pentesting, and software update planning. Pentesting is heavily emphasized in the PCI standards and assessment procedures.
Inspecting server logs provides vivid proof of security events. If you have configured your logging correctly you can detect hacking attempts and trace them. If you are dealing with few servers with little activity, inspecting the logs is straightforward; you can use simple Linux commands such as tail. Digging through large logs from numerous systems requires the use of logging analysis software such as Splunk. Splunk provides an intuitive web interface for fast searching in numerous logs from multiple systems. It can also notify you when it sees certain preconfigured events and help you prevent security disasters. However, knowing which logs to monitor and what to look for requires you to understand each service itself. Because applications and logs are very different, the only general recommendation we can give is to look for abnormal behavior.
The next step in the security audit is to compare and scan the server files. Finding malicious content is not easy because hackers' code can be easily obfuscated, encoded, and encrypted, leaving it hidden from even the most advanced scanners and security products. Also, no matter how experienced a programmer you are, you may still not be able to find a malicious snippet of code hidden within hundreds of thousands of lines of other code. Here you can get help from AIDE, the Advanced Intrusion Detection Environment, which keeps track of all file changes between security audits. However, it requires that you have run it at least once before the actual audit in order to create its database with supposedly uncompromised files. Also, you have to be able to keep track of all the files changes. This may be difficult or impossible if many files on the server change frequently. In that case you can simply try scanning the files for malicious content. For this purpose you use a general-purpose antivirus application such as Kaspersky or a script for a specific purpose, such as this malicious code finder for web files. When performing such file scans bear in mind that they are system resource-intensive. Try to schedule them in hours with light server loads. If necessary you may want to limit your scans' scope to only publicly accessible files.
The next stage of your security scan is the most complex: look for suspicious activities and rootkits on the server. This step is required because no matter how meticulously you scrutinize the logs and files, you can never be certain that your server is secure. It is relatively easy for attackers to hide their traces once they gain access to certain resources. First, check which TCP and UDP ports your server is listening to or has active connections attached to using the command netstat -ntuap. Make sure you are familiar with each program/port pair listed. Don't forget that program names can be arbitrarily set; often attackers use names like "apache2" to fool you into thinking that a genuine service is running. If you have any doubt about a running program, run the command lsof -pXXXX, where XXXX is the pid of the suspicious program. This command lists all open files connected to this pid, including deleted ones.
Under Linux the first 1000 ports are reserved for programs run with superuser privileges (root). Thus attackers with regular account privileges, which means most of them, must run their scripts on port numbered above 1000. It would look extremely suspicious to see a program called apache2 listening on port 6667, run by the user johnb, and listing associated (deleted) files in the directory /tmp.
Checking for suspicious network activity is important because almost every attacker wants to leave a backdoor so he can connect to his victim's machine again. You can also search for any suspicious process, not only network ones, using the command ps auxwf. This command shows information about all running processes and how they have been spawned, including the original files that started them.
If an attacker has gained superuser privileges, you may not be able to see any suspicious activities if he has installed a rootkit. Rootkits can completely alter your environment, changing important executables such as ps, netstat, and who, and loading malicious Linux kernel modules. That's why a security audit always has to include a rootkit scanner such as Rootkit Hunter. This easy-to-use but effective tool reliably ensures the integrity of your system binary files by comparing their MD5 checksums against the genuine corresponding ones. It also scans your server for known loaded kernel-level rootkits.
The above steps are sufficient for the most common Linux server deployments used for tasks such as web, mail, DNS, and databases. However, if your Linux server stores sensitive information about payments or important classified documents, you may decide to go one step further. In a total paranoid security audit you could mount your Linux server hard drive on another machine to inspect all files either manually or using a popular Linux antivirus application. Sometimes mounting the hard drive externally may be your only chance to find out what happened after a heavy server compromise. In some cases attackers' last actions on the server are to render it unusable and destroy it. A popular option for mounting hard drives externally in such cases is to use a live Linux CD such as SystemRescueCd. However, it is not always easy nor practical to mount drives externally – it causes downtime, and you may not have physical access to the server – but it could be more feasible if you use virtual servers and you have access to the storage files.
A final word to the wise: Linux server security audits should not be considered one-time events. Rather, they should be regularly scheduled. Once you begin performing them you will find easier ways to automate some of the tasks. Once your audits show that your Linux server is safe, you can be more confident about your stored data, the uptime of the supported services, and ultimately your business's prosperity.
Allowed tags: <a> link, <b> bold, <i> italics