- 1 Overview
- 2 Ensure That the Software Used to Examine Systems Has Not Been Compromised
- 3 Monitor and Inspect Network Activities
- 4 Monitor and Inspect System Activities
- 5 Inspect Files and Directories for Unexpected Changes
- 6 Investigate Unauthorized Hardware Attached to the Network
- 7 Look for Signs of Unauthorized Access to Physical Resources
- 8 Review Reports of Suspicious System and Network Behavior and Events
- 9 Take Appropriate Actions
- Chapter 6 Checklist
This chapter is from the book
This chapter is from the book
This chapter is from the book
6.4 Monitor and Inspect System Activities
System activities include those associated with system performance, processes, and users. Programs executing on your networked systems typically include a variety of operating system and network services, user-initiated programs, and special-purpose applications such as database services. Every program executing on a system is represented by one or more processes. Each process executes with specific privileges that govern what system resources, programs, and data files it can access, and what it is permitted to do with them. The execution behavior of a process is demonstrated by the operations it performs while running, the manner in which those operations execute, and the system resources it uses while executing. Operations include computations; transactions with files, devices, and other processes; and communications with processes on other systems via your network. User activities include login/logout, authentication and other identification transactions, the processes they execute, and the files they access.
If you are reviewing system activities on a host other than the one being monitored, ensure that the connection between them is secure, as described in Section 2.13
You need to verify that your systems are behaving as expected and that the processes executing on your systems are attributed only to authorized activities by users, administrators, and system functions. Unexpected or anomalous system performance may indicate that an intruder is using the system covertly for unauthorized purposes. The intruder may be attempting to attack other systems within (or external to) your network, or running network sniffer programs. A process that exhibits unexpected behavior may indicate that an intrusion has occurred. Intruders may have disrupted the execution of a program or service, causing it either to fail or to operate in a way other than the user or administrator intended. For example, if intruders successfully disrupt the execution of access-control processes running on a firewall system, they may access your organization's internal network in ways that would normally be blocked by the firewall.
If you permit access to your systems and networks by third parties (vendors, contractors, suppliers, partners, customers, etc.), you must monitor their access to ensure that all their actions are authentic and authorized. This step includes monitoring and inspecting their system activities.
6.4.1 Notify Users
Inform authorized users of your systems about the scope and kinds of monitoring you will be doing and the consequences of unauthorized behavior.
A common method for communicating this message is to present a banner message immediately before user login, as described in Section 6.3.1.
Without the presentation of a banner message or other warning, you probably cannot use log files and other collected data in any action you may choose to take against a user.
6.4.2 Review System Alerts
Review and investigate notifications from system-specific alert mechanisms (such as e-mail, voice mail, or pager messages), including the following:
Users and other administrators, via e-mail or in person
Operating system alert mechanisms
System management software traps
Intrusion detection systems
Custom alert mechanisms from service or application programs (including tools)
6.4.3 Review System Error Reports
These types of notifications are typically produced by the following devices:
Operating system error-reporting mechanisms
Log file filtering tools
Vendor or custom-developed management software
Custom error-reporting mechanisms from service or application programs (including tools)
Often an administrator will be able to configure error reporting at a number of criticality, severity, or priority levels when installing the system, service and application programs, and supporting tools.
6.4.4 Review System Performance Statistics
Statistics are generally produced by vendor or custom performance-monitoring tools. Typical statistics include the following (refer to Section 5.3, Table 5.2):
Total resource use over time—CPU, memory (used, free), disk (used, free)
Status reported by systems and hardware devices such as print queues
Changes in system status, including shutdowns and restarts
File system status (where mounted, free space by partition, open files, biggest file) over time and at specific times
File system warnings (low free space, too many open files, file exceeding allocated size)
Disk counters (input/output, queue lengths) over time and at specific times
Hardware availability (modems, network interface cards, memory)
Performance statistics meaningful for a specific server or host4
Comparison of previous system performance statistics with current statistics
Unexpected shutdowns, reboots, and restarts can indicate the presence of a Trojan horse program that requires a shutdown or restart of a system or service.
Investigate anything that appears anomalous.
6.4.5 Monitor Process Activity and Behavior
The examination of processes is complex, time-consuming, and resource-intensive. The degree to which you are able to identify suspicious processes depends on your knowledge of what processes you normally expect to be executing on a given system and how they should behave.
Due to the large number of processes and their rapidly changing natures, it is impractical for you to monitor them continually yourself. In addition, the amount and value of information that you can gather from a snapshot of currently executing processes may be very limited. This means that you must employ a variety of information-gathering and monitoring mechanisms to help you collect and analyze data associated with processes, and to alert you to suspicious activity.
One common approach with multi-user systems is to set up consoles (or separate terminal windows on workstations) that display the current status of processes and are updated at short intervals. Ideally, these consoles should be hard-wired to the systems for which they are displaying information. With strategic placement of these displays, you can take advantage of the experience of system administrators to notice unexpected activity that may not be picked up by your more immediate alert mechanisms.
Identify any unexpected, unusual, or suspicious process behavior and the possible implications. As a general guideline, you should look for the following:
Missing processes
Extra processes
Unusual process behavior or resource utilization
Processes that have unusual user identification associated with them
Data from log files and other data collection mechanisms will help you to analyze the process behavior, for example (refer to Section 5.3, Table 5.2):
User executing the process
Process start-up time, arguments, file names
Process exit status, time duration, resources consumed
The amount of resources used (CPU, memory, disk, time) by specific processes over time; top "x" resource-consuming processes
System and user processes and services executing at any given time
The means by which each process is normally initiated (administrator, other users, other programs or processes), with what authorization and privileges
Devices used by specific processes
Files currently open by specific processes
Look for processes that are operating in one of the following ways:
Running at unexpected times
Terminating prematurely
Consuming excessive resources (wall clock time, CPU time, memory, disk), which may warn you of an impending DoS condition or the use of a network sniffer
Password cracking, network packet sniffing or any other process not due to normal, authorized activities
Unusually formatted in their output or arguments (for example, on UNIX systems, a process running as ./telnetd instead of /usr/sbin/telnetd)
New, unexpected, or previously disabled, possibly indicating that intruders have installed their own version of a process or service or are running IRC services, web services, FTP services, and so forth to allow them to distribute tools and files they have stolen (such as password files) to other compromised hosts.
Being spawned by inactive user accounts using CPU resources
A terminal process exhibiting abnormal input/output behavior
Without a controlling terminal and executing unusual programs
Unusually large in number
Pay close attention to the processes associated with intrusion detection and other security tools. Intruders regularly compromise these tools to gain greater leverage and information and to generate decoy alerts to distract and waste the time of system administrators.
6.4.6 Monitor User Behavior
Identify any unexpected, unusual, or suspicious user behavior and the possible implications.
Data from log files and other data collection mechanisms will help you to analyze user behavior, for example (refer to Section 5.3, Table 5.2):
Login/logout information (location, time): successful, failed attempts, attempted logins to privileged accounts
Login/logout information on remote access servers that appears in modem logs
Changes in user identity
Changes in authentication status, such as enabling privileges
Failed attempts to access restricted information (such as password files)
Keystroke-monitoring logs
Violations of user quotas
Look for the following types of intrusions and intrusion attempts:
Repeated failed login attempts, including those to privileged accounts
Logins from unusual locations or at unusual times, including unusual or unauthorized attempts to log in via a remote access server
Unusual attempts to change user identity
Unusual processes run by users
Unusual file accesses, including unauthorized attempts to access restricted files
Users logged in for an abnormal length of time (both short and long)
A user executing an unexpected command
A user working from an unusual terminal
If you notice unusual activity associated with particular users, initiate supplemental data collection mechanisms to gather detailed information about their activities. Many multiuser systems provide mechanisms to audit all processes associated with a particular user. Since process accounting logs tend to generate a great deal of information rapidly, you will need to allocate sufficient resources to store the data collected. Similarly, detailed network logging of all activity associated with all the systems accessed by a specific user can be voluminous, and you will need to allocate resources accordingly. Review the newly collected data often (at least daily) and rotate files regularly to minimize the amount of information that you have to analyze at any given time (as described in Section 5.4).
6.4.7 Monitor for the Presence of Network Sniffers
One thing intruders commonly do is to gather information from the traffic on your networks to find user account names, passwords, and other information that may facilitate their ability to gain access to your systems. They do this by breaking into one system on your network and installing and executing a sniffer program. This program collects information about connections established between systems from network data packets as they arrive at or pass by the compromised system. To hide this illicit activity on compromised systems, intruders typically modify log files and replace programs that would reveal the presence of the sniffer program with Trojan horse versions. The substitute programs appear to perform the same functions but exclude information associated with the intruders and their activities. In many documented cases of this type of intrusion, the intruders' activities went unnoticed for a considerable amount of time, during which they collected enough information to gain privileged access to several other systems.
Detecting the presence of distributed network sniffers may not be possible. Some operating systems (but not all, or even most) respond differently to an ICMP echo request when the interface is in promiscuous mode than when it is not, thus providing some indication that something is amiss. Even when this indication is present, however, the computer is under intruder control and will behave as the intruder directs. Without sophisticated analog electronic signaling techniques, it's probably impossible to detect a distributed sniffer externally.
This reality underscores the importance of using verified software to examine your systems (as described in Section 6.2) and the need to verify the integrity of your files (as described in Section 6.5). Unfortunately, intruders can use several sophisticated collections of programs to gain rapid access to systems and "set up shop" to install and execute a sniffer. In such cases the only way you may be able to catch such activity is to use verified software to examine processes on your systems for unexpected behavior (as described in Section 6.4), although this method is not effective against kernel modifications.
Processes associated with a sniffer will typically have transactions with a network interface that has been placed in promiscuous mode, as well as a file or network connection to which the information gathered from network packets is being sent. However, keep in mind that legitimate network monitors and protocol analyzers will set a network interface in promiscuous mode as well.
Network interfaces on most systems normally operate in nonpromiscuous mode, which means that they ignore network packets not explicitly addressed to them. In promiscuous mode, no packets are ignored, that is, all packets that traverse the network segment to which the system is attached are read by its network interface and are accessible to processes executing on that system.
Refer to CERT advisory CA-1994.01, Ongoing Network Monitoring Attacks, at the CERT web site.
6.4.8 Run Network Mapping and Scanning Tools
The purpose of running network mapping and scanning tools is to understand what intruders who use such tools can learn about your networks and systems. We recommend carrying out this task periodically during nonbusiness hours and when you are physically present, because mapping tools can sometimes affect systems in unexpected ways. Eliminate or make invisible (if possible) any aspect of your network topology and system characteristics that you do not want to be known by intruders who use mapping tools.
6.4.9 Run Vulnerability Scanning Tools on All Systems
The purpose of running vulnerability scanning tools on all systems is to check for the presence of known vulnerabilities. We recommend running such tools periodically during nonbusiness hours and when you are physically present, because scanning tools can sometimes affect systems in unexpected ways. Eliminate all vulnerabilities identified by these tools wherever possible. Many of these can be dealt with by updating configuration file settings and installing vendor-provided patches as described in Section 2.4.
Consider using scanning tools that include password analysis as part of their vulnerability assessment. Such analysis may include the identification of weak, nonexistent, or otherwise flawed passwords, such as those that can be determined using brute force or dictionary-based attacks.
Refer to CERT vulnerability notes at the CERT web site and How to Eliminate the Ten Most Critical Internet Security Threats: The Experts' Consensus, Version 1.25 (SANS 00) for a description of some of the more prevalent vulnerabilities.
6.4.10 Policy Considerations
Your organization's networked systems security policy should specify the following:
The need for users to be notified that process and user activities will be monitored and state the objective of such monitoring
The responsibilities and authority of designated systems administrators and security personnel to examine systems, processes, and user activity for unexpected behavior
What forms of unexpected behavior users should watch for and require users to report any such behavior to their designated security officials and system administrators.
What software and data users and administrators are permitted to install, collect, and use, with explicit procedures and conditions for doing so
What programs users and administrators are permitted to execute and under what conditions
6.4.11 Additional Information
If you are reviewing system activities on a host other than the one being monitored, ensure that the connection between them is secure, as described in Section 2.13.
Whenever possible, analyze and correlate data collected from multiple sources, as recommended in the other practices of this chapter. Performing some level of correlation analysis during the intrusion detection process, such as determining when intrusion activity occurring in one part of your systems may be related to activity in another part, will assist you in determining the full extent of any compromise and its characteristics as described in Section 7.2.
Logging information produced by vulnerability patches (updated software that corrects or closes a vulnerability), if provided by the vendor and if turned on, can help identify a pattern in which an intruder exploits more than one vulnerability before gaining access. For example, a failed logged attempt to probe for an old vulnerability (produced by the vulnerability patch) could be followed by a successful probe for a new vulnerability that is not logged. The presence of the vulnerability patch logging information, along with other mechanisms such as integrity checking, could alert you to this type of intruder action.