General Rules for Hardening Linux Server:
Security is a process, not a result. It is a process which is difficult to adopt under normal conditions; the problem is compounded when it spans several job descriptions. All the system level security in the world is rendered useless by insecure web-applications. The converse is also true—programming best practices, such as always verifying user input, are useless when the code is running on a server which hasn’t been properly hardened. Securing forward facing GNU/Linux web servers can seem like a daunting task, but it can be made much easier by breaking the process into manageable portions.
This article will cover installing, configuring and hardening free software web servers and associated software including Apache 2.2.0, MySQL 5.0.18, PHP 5.1.2, Apache-Tomcat 5.5.16 and common Apache modules such as mod_security, mod_ssl, mod_rewrite, mod_proxy and mod_jk. Common security mistakes in web-applications and how to fix them will also be discussed, focusing on PHP and Java environments.
The most common and apt analogy for security is the onion. That is to say it is a layered approach—any one layer is inadequate, the onion is the sum of its layers. With that in mind, this article attempts to bridge the knowledge gap between system administrators and web developers, allowing individuals tasked with security to achieve a layered security solution.
Only a basic understanding of GNU/Linux and common command line tools is assumed.
Note: due to formatting constraints, long lines of code are often broken into several smaller lines using the\ character. This is not a return and when typing in the line you should not hit the enter key, it is just to prevent line wrapping. Output from commands will also be limited to relevant fields, so the output will look slightly different when you run the commands on your system.
Security is a process, not a result
Security at the system level
System level security is one of the most crucial layers in any defense. Hardening at the system level is roughly categorized into network security and file system security.
Network level security can be increased by securing common services such as xinetd (otherwise known as the super server) and OpenSSH, by correctly configuring or disabling them and enabling a firewall (in our case, iptables.
File-System security can be increased by: preventing common avenues of attack, such as root kits; enabling intrusion detections systems (IDS) to verify the integrity of key configuration files; by using tools to detect and remove root kits; and by configuring your logging system so that it will log to a remote host, thereby protecting the integrity of your system logs.
Network security
The first thing you need to do to secure a system from network attacks is find out which processes are listening for connections and on which ports. There are several time tested tools available for this: nmapand netstat.
netstat
The following command will show you which ports are being listened on, the IP address of the listening socket, and which program or PID is associated with the socket (note: running as the super-user or root is necessary for the program field to work properly).
$ netstat -l -n -p -t -u -w
(-l is for listening, -n is for IP information and -p is for program/PID information, -t, -u, -w are fortcp, udp and raw socket connections. By setting these flags, I disable displaying information aboutunix sockets which are not relevant to network security, as they are only used for interprocess communication on the current host.)
The output will look something like this:
Note: Certain columns have been omitted for space
proto Local Address State PID/Program name
tcp 127.0.0.1:8005 LISTEN 4079/java
tcp 0.0.0.0:8009 LISTEN 4079/java
tcp 0.0.0.0:3306 LISTEN 18542/mysqld
tcp 0.0.0.0:80 LISTEN 23736/httpd
tcp 0.0.0.0:8080 LISTEN 4079/java
tcp 0.0.0.0:22 LISTEN 11045/sshd
tcp 0.0.0.0:3128 LISTEN 23283/(squid)
tcp 127.0.0.1:25 LISTEN 24453/master
udp 0.0.0.0:3130 23283/(squid)
udp 0.0.0.0:32870 23283/(squid)
Understanding the output from netstat is pretty simple. The first field is the protocol, and you will notice that when the protocol is udp, there is no state (as obviously udp is stateless unlike tcp). The next interesting field is the Address field. 0.0.0.0:80 means that the server will respond to any IPs on port 80, while 127.0.0.1:80 means that the server is only listening to the loop back device.
nmap
Another tool in our arsenal is nmap, the network mapper. nmap is good for determining what ports and services are available on a server from other machines on the network.
(Note: The default option is -sS. However, when the system being scanned is running a firewall, such asiptables, it won’t work, as firewalls that block icmp traffic will also block the subsequent scan and the results will be meaningless. The -P0 option disables pinging the host before scanning it, The -O (as in “oh” rather than zero) is to enable nmap’s operating system detection via the network stack fingerprint.)
$nmap -P0 -O 10.0.2.10
The output will look something like this:
The 1661 ports scanned but not shown below are in
state: filtered)
PORT STATE SERVICE
22/tcp open ssh
443/tcp closed https
Device type: general purpose
Running: Linux 2.6.X
OS details: Linux 2.6.7 - 2.6.8
Uptime 40.462 days since Mon Dec 26 10:05:57 2005
Now that I know what services are listening on which ports, I can go about securing them. In some cases, the solution will be disabling the unwanted service via inetd; in others, I will use iptables rules to block external access to that port.
In the context of a web server, I would recommended disabling all services managed by inetd (if they aren’t already).
/etc/xinetd.conf (Red Hat): this file usually has some minimalistic configuration of the logging software and then an include statement for all the files under /etc/xinetd.d, which are configuration files for each service run through the super server.
/etc/inetd.conf (Debian): Debian has a much simpler configuration layout—one simple file/etc/inetd.conf containing one line for each service managed by inetd.
iptables
The venerable iptables has been the standard Linux firewall since the 2.4 kernel. The kernels that come with Red Hat and Debian have the proper modules enabled; however, on Debian systems you may need to install the iptables user land tools. Configuring iptables is fairly simple: iptables has chains, rules and targets. iptables has three built in chains: FORWARD, INPUT, and OUTPUT. To create an effective firewall I will append rules to chains that will be matched by connection type, source or destination address or state. In more advanced configurations, it is favorable to create custom chains and then reference them in the default chains; but, to demonstrate the basic principles, I am just going to append rules to the three default chains. When a connection is being matched against the configured rules, each rule is checked. If it matches, it is executed, if not, the next rule is tested. As such, the rules allowing traffic should be appended first, and the very last line in any chain should be a deny rule. This is the most secure firewall configuration, where everything is dropped except the explicitly allowed connections.
If you use Debian, run:
$apt-get install iptables ( to install iptables )
$apt-cache search iptables ( to search for packages related to iptables)
To get started with iptables I will list the current rule set using the following command:
(Note: Output has been modified due to formatting constraints.)
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all anywhere anywhere \
state RELATED,ESTABLISHED
Chain FORWARD (policy ACCEPT)
target prot opt source destination
ACCEPT all anywhere anywhere \
state RELATED,ESTABLISHED
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
DROP tcp anywhere anywhere \
tcp dpt:ssh
The partial listing above shows rules that allow incoming traffic that isn’t new; that is to say: the connection has been established from inside the network. IP forwarding follows the same rule, and using ssh to connect out to other hosts is blocked.
The flush command with no options will flush all rules; if a chain is passed, all rules in that chain will be flushed. I’ll flush all rules and begin configuring the firewall.
$iptables -F
or
$iptables -F INPUT
$iptables -F FORWARD
$iptables -F OUTPUT
Next, I am going to append the rules to the appropriate chain. A high level overview of the firewall will be the following:
- Allow outgoing connections initiated from the host
- Allow inbound ssh connections on port 2
- Allow inbound http connections on port 80
- Allow inbound https connections on port 443
- Block outbound ssh connections
- Block everything else
# Enable stateful filtering allowing connections
# initiated on host be allowed.
$iptables -A INPUT -m state --state \
RELATED,ESTABLISHED -j ACCEPT
$iptables -A OUTPUT -m state --state \
NEW,RELATED,ESTABLISHED -j ACCEPT
# Allow Incoming SSH, HTTP, HTTPS
$iptables -A INPUT -p tcp -m tcp \
--dport 22 -j ACCEPT
$iptables -A INPUT -p tcp -m tcp \
--dport 80 -j ACCEPT
$iptables -A INPUT -p tcp -m tcp \
--dport 443 -j ACCEPT
# Allow Everything from the local host
$iptables -A INPUT -s 127.0.0.1 -j ACCEPT
# Block Outgoing SSH connections
$iptables -A OUTPUT -p tcp -m tcp \
--dport 22 -j DROP
# Block Everything else
$iptables -A INPUT -j DROP
$iptables -A FORWARD -j DROP
To save the changes I have made to the firewall rules I use the iptables-save command:
$iptables-save > /root/firewall
Later if I wanted to restore my saved rules I would run the iptables-restore command:
$iptables-restore -c /root/firewall
It’s a very good idea to have these rules applied at boot time; check your distribution’s documentation for this. In general, on Debian systems the network configuration scripts can be used for this, and on Red-Hat systems a startup script in /etc/init.d is appropriate.
Changing the default port that `OpenSSH` listens on is a good way to avoid brute force attacks
Hardening SSH
The OpenSSH package comes installed by default on most distributions. The default configuration on most distributions is pretty lax and favors functionality over security. Allowing root logins, listening on all IPs on port 22, and allowing all system accounts to ssh-in are all potential security holes.
Edit /etc/ssh/sshd_config in your favorite editor and change the following lines.
# ListenAddress defines the IP address ssh will
# listen on
#ListenAddress 0.0.0.0 -> ListenAddress 10.0.2.10
#Only accept SSH protocol 2 connections
#Protocol 2,1 -> Protocol 2
#Disable root login
PermitRootLogin yes -> PermitRootLogin no
#Disable allowing all system accounts to ssh in,
# only allow certain users (space delimited)
AllowUsers userName1 userName2 userName3
# Change Default port
Port 22 -> Port 2200
After making the changes, restart the SSH server for the changes to take affect:
$ /etc/init.d/ssh restart
Partition for security
File system security
The UNIX file system has several standard directories: /, /tmp, /var, /usr and /home. The two that present the weakest links for a variety of attacks are /tmp and /var. The two most common attacks are: “Denial of Service”, by causing the root partition to fill up with logs or other junk (assuming all these directories are mounted on one partition); and running rootkits from the /tmp directory.
One solution to file system Denial of Service attacks is to have these directories mounted on their own partitions, this will prevent the / file system from filling up and stop that avenue of attack.
Rootkits typically write to the /tmp directory and then attempt to run from /tmp. A crafty way to prevent this is to mount the /tmp directory on a separate partition with the noexec, nodev, and nosuidoptions enabled. This prevents binaries from being executed under /tmp, disables any binary to be suidroot, and disables any block or character devices from being created under /tmp.
Edit /etc/fstab with your favorite editor, find the line corresponding to /tmp and change it to look like this one.
/dev/hda2 /tmp ext3 nodev,nosuid, noexec 0 0
Wikipedia [6] defines rootkits as a set of software tools frequently used by a third party (usually an intruder) after gaining access to a computer system. This translates to custom versions of ps that won’t list the irc server the attacker installed, or a custom version of ls that doesn’t show certain files. Tools likechkrootkit must be run in combination with IDS systems like fcheck to prevent the successful deployment of rootkits.
chkrootkit is very simple to run, and doesn’t require any installation or configuration.
It’s a good idea to run chkrootkit at regular intervals, see the script below used by fcheck for inspiration.
# Use the wget utility to download the latest
# version of chkrootkit
wget ftp://ftp.pangeia.com.br/pub/seg/pac/chkrootkit.tar.gz
tar -xzvf chkrootkit.tar.gz
cd chkrootkit-version (whatever version is)
./chkrootkit
The next layer of file system security is maintaining and verifying the integrity of configuration files that are typically located under /etc. Intrusion Detection Systems (IDS) allow us to create cryptographic identifiers of important configuration files and store them in a database. They are then periodically re-created and verified against those stored in the database. If there is a mis-match, the file has been changed, you know your system integrity has been violated and which aspects of it are affected. Two well known IDS packages are tripwire and fcheck, which work equally well. However, fcheck has a much simpler configuration and installation process, which is why I favored it for this article.
fcheck
Download fcheck (see resources) and unpack it. fcheck is a cross-platform Perl script which runs on UNIX and Windows systems (as long as they have Perl installed).
$mkdir /usr/local/fcheck
$cp fcheck /usr/local/fcheck
$cp fcheck.cfg /usr/local/fcheck
Edit /usr/local/fcheck/fcheck.cfg with your favorite editor and change the following values:Directory, FileTyper, Database, Logger, TimeZone, and Signature.
# Directories that will be monitored
# if there is a trailing / it will be recursive
Directory = /etc/
Directory = /bin/
Directory = /sbin/
Directory = /lib/
Directory = /usr/bin/
Directory = /usr/sbin/
Directory = /usr/lib/
TimeZone = PST8PDT # For Pacific Standard
# Database of file signatures
DataBase = /usr/local/fcheck/sol.dbf
Logger = /usr/bin/logger -t fcheck
# Utility to determin file type
FileTyper = /bin/file
# What to use to create signatures Database of
# file signatures
$Signature = /usr/bin/md5sum#
DataBase = /usr/local/fcheck/sol.dbf
Logger = /usr/bin/logger -tfcheck
# Utility to determin file type
FileTyper = /bin/file
Also edit the fcheck script and change the path of the configuration file to/usr/local/fcheck/fcheck.cfg
Then run fcheck for the first time to create the baseline database.
# Options explained:
# c create the database
# a is for all
# d is to monitor directory creation
# s is to create signatures for all files
# x is for extended permissions monitoring
$ ./fcheck -cadsx
To test that everything has been setup correctly run the following commands and fcheck should alert you to the difference.
$ touch /etc/FOO
$ ./fcheck -adsx
fcheck should display some information about /etc/FOO. $rm /etc/FOO will prevent future messages.
Next, create a short shell script that will be run periodically by cron and check for changes. Open your favorite editor and create /usr/local/bin/fcheck_script.
When using the `cron` utility lookout for _symlink attacks_
#!/bin/bash
# Use mktemp instead of $$ to prevent sym-link attacks
FCHECK_LOG=`mktemp`
# Grep for any changes
/usr/local/fcheck/fcheck -adsx \
| grep -Ev ^PROGRESS: |^STATUS:^$ > $FCHECK_LOG
# If there were any changes email the sys-admin
if [-s $FCHECK_LOG ] then
/usr/bin/mail -s fcheck \
`hostname` youremail@yourprovider.com < \
$FCHECK_LOG
/bin/rm $FCHECK_LOG
fi
The cron utility will be used to run periodic checks of the file-system and will compare it to the baseline database. The following command will edit root’s crontab:
$ crontab -e
# Add this line to run the script every 15 minutes
# using nice lower priority when the system load
# is high.
*/15 * * * * nice /usr/local/bin/fcheck_script > \
/dev/null
Symlink Attacks
Side Note: Symlink Attacks running an IDS package usually involve running a script at a pre-configured time using the cron utility. This opens up systems to symlink attacks. Symlink Attacks rely on the attacker knowing that a certain file is going to be created at a certain time with a certain name. A common shell scripting technique that generates some randomness is the use of $$, which is the PID of the running script. However, this is vulnerable to Symlink Attacks because most PIDs are below 35K and most file systems can have 35K files. The correct technique is the use of mktemp, which is a truly random file name.
http://www.freesoftwaremagazine.com/articles/hardening_linux
http://www.frank4dd.com/howto/apache/apache-hardening-top-ten.htm
http://itfanatic.com/?q=node/52
http://security.stackexchange.com/questions/9948/how-to-secure-a-dedicated-linux-server-running-a-lamp-stack-for-commercial-e-com
