- : Executing a Policy
- Security Incident Response
- Computer Security Incident Response Teams
- Preparing for Incident Response
- Management of Security by Teams
- Execution of an Incident Response
- Evaluation of a Security Incident
- Containing the Incident
- Eradicating the Incident
- Recovering From an Incident
- Article Series
- About the Author
- Acknowledgements
- References
- Ordering Sun Documents
- Accessing Sun Documentation Online
Like this article? We recommend
Like this article? We recommend
Like this article? We recommend
Eradicating the Incident
Eradication essentially means ending the intruder's attack on the customer's system. It can be achieved by turning off the affected system and network (if known) and/or disconnecting from the Internet, as well as from the rest of the internal network at the customer site. Secondly, eradication involves restoring systems to a secure baseline configuration. This step should not be underestimated by the VCSIRT or its counterparts acting on behalf of the VCSIRT at the customer site. A major incident, or multiple almost simultaneous incidents, can cripple a large site with hundreds or thousands of computers or computer-based equipment. Sometimes, the best way is to completely erase and rebuild the system from scratch. But, that is not always practical from the following points of view:
The time and complexities involved in performing a rebuild
The trust involved in the components used for restoration by the VCSIRT or their representatives at the affected constituent's site
The procedures that must be executed to preserve the evidence of the intrusion or attack before valuable forensic data left by the attackers on hard drives or in core memory is erased. (This topic, along with chain-of-custody aspects, will be discussed in detail in the next article of this series.)
According to ITU-T X.509, Section 3.3.54, trust is defined as follows: "Generally an entity can be said to trust a second entity when the first entity makes the assumption that the second entity will behave exactly as the first entity expects."
When doing a complete rebuild, the constituent must be advised of trusted sources (for example, CD-ROMs or trustworthy locations providing guarantees of downloaded contents through methods such as message digests—for example, MD5 checksums). For more information on message digests, refer to:
http://www.sun.com/blueprints/tools/fingerprint_licence.html
In addition, the constituent should be guided in the use of state-of-the-art software tools and a clean backup to remove or mitigate damage and to keep a safe copy for future use when the system usage is resumed.
But, what assurance is there that the backup would be clean when the system or network resumes its operation? Every site does not have the best tools, expertise, or time at hand to remove malicious code (such as worms and viruses) and to research and patch vulnerabilities. That is why it is important that the organization assisting its constituent customers provides the best possible tool set or proactively makes sure the customers have adequate tools to perform recovery from an incident. Otherwise, the customers might be setting themselves up for a future incident or for a repeat incident of the same kind.
Common Indicators of UNIX System Intrusions
In spite of repeated publicity at several well-known Web sites by the security coordinating industry organizations, even the most commonly known vulnerabilities are not addressed by a large percentage of sites. The following list contains common indicators of UNIX_ system intrusions. The list is not all-inclusive. Attempted or successful intrusions might not have left obvious evidence. Use all resources available to analyze suspicious or suspected activities.
Binaries
/bin/df
/bin/ls (to hide logs and files that can provide clues to intrusions)
/bin/login and /sbin/login (to provide login backdoors)
/bin/ps (to hide processes that provide online clues of hacker processes—for example, password-cracking utilities)
/usr/etc/in.telnetd (to provide network backdoors—for instance, trojan-horse versions of Telnet that provide a root shell when a particular value is set for TERM)
crontab (to hide scheduled jobs that steal passwords or cleanup logs)
/usr/sbin/ifconfig (to provide a trojan-horse ifconfig file that hides the use of the promiscuous mode of network interfaces, which is used by sniffers to identify passwords)
/usr/ucb/cc
/usr/ucb/netstat (to hide unauthorized connections)
File modifications
.cshrc
.forward
.login
.profile
/.rhosts
/bin/.rhosts
/etc/hosts.equiv
/etc/group
/etc/passwd
Hidden directories
/etc/tmp
/tmp
/var/tmp
/usr/lib/cron
/usr/spool
Poor passwords
Weak passwords (that is, nouns or words used from dictionaries) should not be used as passwords. There are well-known rules of password construction (for example, using combinations of alphabetic characters—upper and lower case, 0-9 digits, and punctuation marks).
Accounts with default passwords should not be used (for example, vendor supplied ones). Scan your password file for extra UID 0 accounts.
Reusable passwords should not be used. At the very least, a single password should not be used for multiple accounts. It is common for intruders (who might be camped out on the internal network) to harvest (as in, clear-text) user name and password information using packet sniffers from a subnet, a local physical network, or a wireless network.
Use of TFTP (Trivial File Transfer Protocol) to obtain password files
Vulnerabilities in sendmail(8)
Misconfigured anonymous FTP
Inappropriate network configuration file entries
Inappropriate secure settings in /etc/ttys and /etc/ttytab
Inappropriate entries in /etc/aliases or /usr/lib/aliases
Old versions of system software
Use of setuid shell scripts
Inappropriate export settings
Vulnerable protocols and settings
Inappropriate file and directory protections
Tape archive: tar -flags device file_system_to_be_dumped
dump (the primary backup program): dump -flags device file_system_to_be_dumped
Device-to-device copy: dd if=input_file output_file
-
CERT/CC at: ftp://info.cert.org:/pub/tools
-
DFN-CERT at: ftp://ftp.cert.dfn.de/pub/tools/
-
Purdue University's Computer Operations, Audit, and Security Tools (COAST) at: coast.cs.purdue.edu:/pub/tools
-
Sun BluePrints™ Scripts and Tools Web site at: http://www.sun.com/blueprints/tools/
If an intruder has gained unauthorized access to your system, quite often the intruder will try to replace or alter one or more of the binaries in the following list. This is a sample list only. The goal of the hacker is mentioned in parentheses for some of the example files:
The intruder might attempt to modify one or more of the following critical files:
Intruders often attempt to hide files in one or more of the following directories:
Common UNIX System Configuration Issues
The following list contains commonly known and exploited UNIX configuration issues that translate into a not-to-do list on rebuild or on resumption of operations. (These are more or less self-explanatory, having been announced in vulnerability notices over the years, and available at public sites in various forms, but a brief description is provided here.)
Disable tftpd. If you need to have it, design restricted access. If your constituent suspects that passwords have been stolen, then of course, change all of them.
Numerous vulnerabilities have been found in sendmail. The sendmail restricted shell (smrsh) controls the way mail messages can interact with the operating system. mail.local can be used to control the way /bin/mail is used on the system.
You should not use your system's standard password file or group file as the password file or group file for FTP. Also, the anonymous FTP root directory and its two subdirectories should not be owned by FTP.
Watch out for this common vendor mistake (/etc/hosts.equiv with a + sign entry). The + sign indicates that your system trusts all systems.
The only terminal that should be set to secure should be the console.
Remove entries such as uudecode, if they are not yours. These entries are setups for hackers.
When to patch is a common issue for system or network administrators, generally due to lack of time or resources. Thus, this type of faltering opens doors for attackers. It is argued by theorists that ideally, the best time to patch is not when the patch first comes out from the vendor because it could have new bugs or there could be a vendor recall due to other business reasons. However, the wait introduces an element of risk.
setuid root shell scripts are a potential security risk.
Use showmount to verify /etc/exports files. Watch out for examples, self-reference of NFS server, export file systems to hosts (with fully qualified host names) that are known to require them. For further details, refer to:
ftp://infor.cert.org/pub/cert_advisories/CA-94:15:NFS.Vulnerabilities
Consider filtering certain TCP/IP services at your site boundary firewall and router. For further details, refer to:
ftp://info.cert.org/pub/tech_tips/packet_filtering
Systems with these can be a hacker's paradise. Use the vendor's system documentation to verify that system files and directories have correct protections and ownership (in particular, the root, /, and /etc directories).
Many guidelines exist for recovery from root compromises. For more information refer to CERT/CC:
http://www.cert.org/tech_tips/intruder_detection_checklist.html#B2
While following the guidelines provided at public sites, such as above, VCSIRTs must keep their parent organization's policy in perspective.
Capturing Evidence Before Cleaning Up
The compromised system cannot be trusted under any circumstance. System binaries, configuration files, and even the system's kernel might have been modified by the attacker(s). For example, after compromising a machine, attackers, especially in UNIX systems, install programs called root-kits that allow them to regain access and to cover their tracks, as in removing entries from system logs. There might be pressure from the constituent's management to get the system back online, but you must follow a process (defined in the policy) to preserve the data on the compromised system for a later analysis, which can be time-consuming. Great care must be taken to collect all of this necessary information because it can be lost in the process of cleaning up the system's hard drives and core memory. If bogus files and attacker's toolkits have been created or left behind by an intruder, archive them before deleting them.
All further analysis should be done on a trusted copy (which was previously built and physically secured), not on the original compromised system. Many times, it might be deemed necessary to leave the compromised system online to collect all of the proof of an attack, but a note of caution here is that there is a possibility that the attacker could regain control, again and again, while you are recovering the system, handing you setback after setback. On the other hand, if you have support from the constituent's management team, and you are a forensic expert with the intention to capture the attacker by learning his or her behavior, then this is an opportunity. (This topic will be discussed in detail in the next article.)
There are three types of tools (or operating system commands) available in UNIX and Linux for backups:
For details, check the man pages on your UNIX or Linux system. Note that in most Linux systems, a dash (-) does not appear before any flag. Making a full backup of the victim system immediately when it appears that an incident has occurred is very important. The dd tool can read files that are submitted to it as inputs on a block-by-block basis. Thus, it can capture data, such as deleted blocks, that tar and dump cannot. For detailed guidelines on the use of the dd tool, refer to:
http://www.crazytrain.com/dd.html
Never reuse the system backup of a compromised system because it might reintroduce a bug into the production environment. Later, you might have to reload certain portions of the saved environment, but anything you get from the vendor's distribution media (for example, operating system and associated system binaries) should not be restored from the backed-up disks during the recovery phase.
Using State-of-the-Art Software Tools
Software tools, such as antivirus software, should be available within an enterprise to eradicate damage. A number of public sites are available to download freeware tools. However, care must be taken not to accidentally use tainted software. We could provide a long list, but undoubtedly, many of the tools will be surpassed or become obsolete in the course of time. A sad fact is that there are few security conscious applications currently available. Primarily, this is caused by the need for a sound security infrastructure that must be put in place first for most tools to run on securely. The following list contains a few locations. Be sure to use a well-known mirror of these sites:
The netcat tool is an extremely useful utility for transferring information over TCP or UDP. Be certain that your VCSIRT has the authority to install and use netcat on your customer's network. For details on using netcat, refer to the following sites:
Copying images from compromised systems to the safe (that is, trusted) system can be accomplished with netcat listening on the trusted system, while running trusted dd from the vendor's CD-ROM that is mounted on the compromised system. For analyzing the UNIX system, the recommended tool is The Coroner's Toolkit (TCT). TCT is a suite of tools. For more information, refer to: http://www.porcupine.org/forensics