Like this article? We recommend
Like this article? We recommend
Like this article? We recommend
Looking for the Invisible
After using the RPM trick, let's say that files like netstat and ls actually were modified. The question that followed is fairly obvious: "What now?"
You have a fair number of options. Depending on the importance of the system, I will usually recommend taking a backup of the user directories, password, and other critical system files, and rebuilding the system without these files, using the backup as a reference for the new system. I won't just copy those files back. Our cracker may have hidden things in legitimate places, and we don't want to let him back in quite that easily.
You can also leave the system alone, tie down the host access with TCP wrappers, shutting down nonessential services, and replacing affected packages. Starting clean is important, but we don't always have that luxury—not immediately, anyway. If you discover that your procps or net-tools package has been modified by a cracker, the first thing to do is to reinstall the package. Since that package may have been the hole through which your cracker entered, it is usually a good idea to get the latest build from your vendor (Red Hat, Caldera, Debian, and so on). For the truly paranoid, the fact is that once a cracker has access to your system, they can replace anything, including the very files we use to track down the damage. Like the Shaolin priests in the old TV series "Kung-Fu," the cracker succeeds by being invisible.
Now, let's have a look at those invisible things.
Here is a real-life example. After a cracker attack, the machine was tied down, TCP wrappers were installed, and all affected packages were replaced. It was time to scope out the damage while keeping a close eye on the logs for repeated attempts at break-in. Looking at the /etc/passwd file, I noticed a user that did not belong on the system, jon. It looked like a normal passwd entry and did not have root privs. With several users on this machine, our cracker hid nicely in the passwd list.
When I went to his home directory (/home/jon) and did an ls -l, all I got was this:
. .. .. .bashrc .bash_history .screenrc emech.tar.gz
Other than a file called emech.tar.gz, things did not look that strange. Could that be all that was wrong? With a closer look, though, you'll notice that there are two .. directories (pointers to the previous directory in your filesystem hierarchy). That's strange. However, if I change directory to .. with cd .., I just wind up in the /home directory. What's up?
What's up is that there is an extra space after the second dot-dot. I can find this out like this:
# cd /home/jon # echo .* | cat -v . .. .. .bashrc .bash_history .screenrc emech.tar.gz
Look very closely. Notice how each item is separated by only one space. Now look between the second dot-dot and .bashrc. There are actually two spaces, which means the directory is actually "dot-dot-space." To get into that directory and have a look around, I do this:
# cd ".. "
Now an ls shows me all this fun stuff:
randfiles mech.set mech.pid checkmech cpu.memory mech.help mech.usage mech mech.levels emech.users psdevtab
That's interesting. Let's see if jon has any more files hidden around the disk. Using the find command again, I specify a search for files belonging only to this user ID.
# find / -user jon -print
Aside from what is in the /home/jon directory, I get this partial list:
/usr/local/bin/.httpd /tmp/cl /tmp/.l/bcast /tmp/.l/.l /tmp/.l/imapd /tmp/.l/log /tmp/.l/pscan /tmp/.l/pscan.c /tmp/.l/rpc /tmp/.l/slice2 /tmp/.l/sniffer /tmp/.l/sxploit /tmp/.l/thc /tmp/.l/ufs.c
Looking a bit more interesting, isn't it? Sniffers. Port scanners. Our cracker was making quite a home for himself. Furthermore, we discovered two other users coming from different hosts with their own files. Our cracker was either operating from different locations with different IDs or he had friends.
In doing this search, there were even files belonging to this cracker in legitimate user directories, including one very scary file, something called tcp.log. This file was several hundred lines long and contained every telnet and ftp login that had come to and from the machine. Every one! Aside from telling the person whose machine had been broken into that they should rebuild the whole thing from scratch, I also told them to change each and every password, not only on this system but on every system they have access to.
Here's the scoop. Part of the information your cracker collects is a list of logins and passwords you use on other systems. Why? So they have an easier time breaking into someone else's system. Every system you have been accessing while your cracker has had access to your system is at risk. You should contact the system administrators of those other systems and inform them of the risk they face. The flip side is that someone logging into your system on a regular basis whose system had been hacked may have give the cracker a valid login and password on your system. Spooky, huh?
Here are a few examples to help you search for the hidden and dangerous. For starters, check the user directories for suid or guid files. These are programs that have an s instead of an x when you do an ls. For instance, an ls -l on /usr/bin/passwd returns this information:
-r-s--x--x 1 root root 10704 Apr 14 1999 /usr/bin/passwd
The s in the fourth position means that the passwd program acts as root when it is being executed. This is necessary in order to allow users to change their passwords. The second x is simply an x, but an s in this position would mean that any user in that group would act as that group. Programs that can act as a specific user or group are not a bad thing—usually. That said, for the most part, no regular (nonadministrative) user needs to have root-suid files in their home directories. Look for them this way. The command assumes that your users are created in the /home directory.
# find /home -perm -4000 -o -perm -2000 -print
What else can we do? Since we want to speed up the process of finding programs and files left behind by our cracker, a quick way to look for hidden directories would be good. This command will show you those. It will also show you things like .kde and so on, but you'll also find things like dot-dot-space and dot-dot-dot, perfect hidey-holes for your cracker.
# find / -type d -name ".*" -print
The -type d option means to list directories only. This can be a big list, but it is certainly a smaller one than you would get if you just walked through every file and directory on the system. What's nice here is that your proper dot and dot-dot directories (. being the current directory and .. being the parent directory) do not show up in this list. If you see a dot-dot, it will have some other hidden character following it.
Let's sum up. Blowing away everything on your cracked system and starting over is a quick-and-dirty approach that lets you create a properly secure system right from scratch. Eventually, this is what you should probably do anyhow. If your system must be up, using a new box and making that your new production system is probably the next best bet, but providing a brand new system while you investigate the damage to the old one can be costly. PCs are inexpensive, but not everybody is ready to shell out a few thousand to bring another system online. The catch is this—your cracker has left a wealth of information behind, information you may need. Getting rid of that information is a bit like getting rid of the evidence. It's tough to do an investigation without evidence. Weigh the costs of either decision, and then act. But do act.