Identifying and Investigating Network-Bound Software in a Linux Server

In this article we will learn how to correlate open ports with software running in a Linux server and understand why this knowledge is critical to operating and maintaining a secure environment. When a server is compromised, the result is not always as rash as complete data loss. Often the hacker will use the compromised host to perpetrate his primary goal, which is maintaining anonymity. One method to achieve this is install and operate software which proxies network traffic. Due to this, the ability to generate a list of network-bound software and audit each is important.

This information is not limited to analyzing compromised or servers with security issues, as understanding what software running in your server that accepts input over the network (internet) is vital to keeping your server secure.

After ssh’ing to our server as root, lets check which software is listening on tcp sockets.

# netstat -alntp | grep LIST
tcp 0 0* LISTEN 9653/sshd
tcp 0 0* LISTEN 8118/mysqld
tcp 0 0* LISTEN 5266/httpd
tcp 0 0* LISTEN 15404/openvpn
tcp 0 0* LISTEN 10065/perl
tcp 0 0* LISTEN 5266/httpd
tcp 0 0* LISTEN 23952/privoxy
tcp 0 0* LISTEN 9301/exim4
tcp 0 0* LISTEN 9664/tor

Each row represents an open socket, possibly to the internet. The fourth column tell us the IP address and port on our server the software is listening for connections on. means all IP addresses. If the server does not have a firewall, one can assume that packets from the internet will be processed by the software listening on the appropriate port. Even with a firewall, it is worthwhile to ensure that software is secure. This is because software firewalls can be disabled/malfunction and hardware firewalls can be removed from the network without notice. The last column is in the format X/Y. X is the process ID or PID of the network-bound software. Y represents the name of the running program (this “name” is controllable by the program itself, and thus cannot be relied upon to know what that program actually is). It is critical that you are able to correlate an open socket in your server with a program (PID) in your Linux server.

The first row tells us that “sshd” is running at PID 9653 and listening on IP address port 322. This means tcp connections to port 322 are handled by “sshd”. We can see that “httpd” is listening on ports 80 and 18081. The first port makes sense, this is the default port for http traffic. Port 18081 is a little strange, as it is an uncommon port number. To verify the authenticity of PID 5266, we can start by running this command to determine the absolute path to the binary which is network bound:

# ls -al /proc/5266/exe
lrwxrwxrwx 1 root root 0 Jun 24 20:34 /proc/5266/exe -> /usr/sbin/httpd

This tells us “httpd” is actually /usr/sbin/httpd. This makes sense because this is the normal path and name of the Apache webserver in CentOS. To verify that /usr/sbin/httpd is the true Apache binary, we can run two RPM commands:

# rpm -qf /usr/sbin/httpd
# rpm -V httpd-2.2.3-31.el5.centos.2
S.5….T c /etc/httpd/conf/httpd.conf

The first command uses flags “-qf” to query a file; the output is the package which the file belongs to. We then use the “-V” flag to verify the authenticity of that package in the system. The output consists of any modified files. In this case, we learn that httpd.conf has been modified. This makes sense and is common, as the default apache configuration is not very useful for most organizations.

If /usr/sbin/httpd had been replaced with another binary, we see an immediate sign that the server could be compromised. We will simulate this by placing another binary in place of httpd.

# mv /usr/sbin/httpd /usr/sbin/httpd.real
# cp /bin/ls /usr/sbin/httpd
# rpm -V httpd-2.2.3-31.el5.centos.2
S.5….T c /etc/httpd/conf/httpd.conf
S.5….T /usr/sbin/httpd

In this case, /usr/sbin/httpd fails in three areas (periods represent passed tests):

S is the file size.
5 is the MD5 checksum of the file.
T is the modification time of the file.
We can see that it makes sense for /etc/httpd/conf/httpd.conf to fail S and T, since the file was modified (modification time and file size changed). A failed md5 checksum in this case tells us that /usr/sbin/httpd is not the file which Centos provided us with.

Going back to our netstat report, we find this open socket:

tcp 0 0* LISTEN 5266/httpd

We will now check out this “httpd”:

# ls -al /proc/5266/exe
lrwxrwxrwx 1 root root 0 Jun 25 05:29 /proc/20424/exe -> /tmp/.var/lib/httpd

This is a rogue process, as “programs” should not be stored in /tmp in any normal Linux server. This file should not belong to any package, as rpm can verify for us:

#rpm -qf /tmp/.var/lib/httpd
file /tmp/.var/lib/httpd is not owned by any package

At this point, we know there is some insecurity in this server, as sockets are open by software which we did not install. Tracing the source of this rogue network program is another topic. Deleting the file may solve the immediate issue but rest assured the file and process will return without solving the underlying insecurity.

We should continue to audit each network bound process in our server. PIDs 23952 and 9664 are of particular interest based on the process names. Network connections can also be established via UDP sockets. To generate a list of udp ports bound to programs running in your server, we can use command:

# netstat -alnp | grep udp
udp 0 0* 18443/dnscache
udp 0 0* 19029/named

The same procedure in identifying the responsible binary via the PID can be used.

When provisioning a new server, it is critical to understand and research each open socket. Disabling unnecessary network bound software is critical in limiting the entry points to the server, thus increasing security. After we identify each piece of software which may be exploited, we should check the version and configuration of that software to ensure it is not vulnerable to any known security issues.

This entry was posted in Software and tagged . Bookmark the permalink.