18.3 Drop the Modems, Network, Printers, and System
Once you have detected that your system has been broken into, you must decide how much data to gather on the intrusion before stopping further damage. It is assumed that you already have gathered as much data as you dare by this point. Now you need to get the intruder out of your system. There are two parts to this. The first is preventing him from accessing your system, and I will address that here. (Later you need to remove what he has left behind in the way of compromised programs and plug security holes.)
Many system administrators forget that the most effective way to throw the intruder out is to sever the connections between the computer and the outside world. For most, this means disconnection from the network and modems.
You should plan in advance for your response to an intrusion because you will want to act quickly when an intrusion is discovered. A formal Security Procedures or Intrusion Response manual is recommended. Writing this document in advance is valuable because in the "heat of battle" you might not have the luxury of time nor be able to think as clearly.
This is why every pilot carries an Emergency Procedures document on his aircraft that lists the responses to common emergencies that have been thought out carefully in advance and are based on past experiences. On your next airline flight, on the way out you might ask the flight crew to show you theirs. It will be instructional.
In many cases, the fast way to sever connectivity is to disconnect the modems from either the phone lines or electrical power. For a small setup, simply unplugging the phone cable from the modem or phone jack will do fine. I do this when I am under attack. For larger setups, having all the modems' power plugs in one or two power strips or UPS (Uninterruptible Power Supply) units will allow throwing one or two power switches to turn them off, this being easier than unplugging lots of phone cables and then later trying to figure out which one went where. (Having two sets of power strips or UPS units provides redundancy.)
If the intruder might have gotten in through your LAN, simply unplug the network cable from the computer. I recommend this solution so that you do not disrupt the rest of the network. If you have any local users through serial connections who could be the culprit, you might need to unplug these cables. Keep in mind that it is likely that the intruder has broken into other of your systems too, particularly if they are configured similarly. If you think that this is likely, it might be better to disconnect your entire network from the Internet.
Many SysAdmins forget that the fastest way to shut out intruders is to shut down the system or take it to single-user mode. Once you capture evidence of the break-in, either of these is strongly preferred. The advantage of first dropping the modems or network is that it prevents most of the possible further harm to your system while allowing you to see what processes the cracker left running. Seeing these processes, obviously, is important to tracking down the cracker's methods, damage, and origin.
Now that you have collected all the information that you can about the intruder's current connections, it is time to shut the system down and boot from a disk or tape that is known to have no Trojan horses. An orderly shutdown might alert any Trojan horses that you have missed. It is hard to be confident that you have detected and killed all of them. Because of this, it might be better to stop the system abruptly.
First try to close any database operations becaues these can be delicate. Then issue the sync command from a nonprivileged account and wait two seconds. Then press the computer's reset button or interrupt power. The slight risk of file system corruption probably is less than the risk of alerting a Trojan horse that might destroy the entire file system or send e-mail to the cracker alerting him that he might have been discovered.
Before coming up multi-user, inspect /var/spool/mqueue for possible cracker-generated e-mail that he might be using to alert himself that he has been discovered. If you suspect that he could be using an idle account, issue the command
ls -ltr /var/spool/mail
and observe which accounts have the most recent e-mail. Are any of these accounts unused or accounts of people on vacation? Certainly, someone could be receiving e-mail while on vacation. Personal accounts' e-mail should not be looked at unless the "owner" is unavailable and only with written permission from management.
There might even be laws in your jurisdiction forbidding this on the basis of "privacy." Having a written policy in advance that "all e-mail and disk files are subject to inspection as needed for system administration" might grant you the authority. This is another issue to work out with management, Human Resources (Personnel), and the Legal Department in advance.
After shutting the system down it is time to switch to Auxiliary Control. Setting this up was discussed in "Switch to Auxiliary Control (Hot Backups)" on page 437. If you do not have it then use Tripwire or the tar technique to find what was altered and correct. Failing this, it is time for backup tapes or CD-RWs.