Solaris Operating Environment Network Settings for Security: Updated for Solaris 9 Operating Environment

• By Sun Microsystems
• Sep 12, 2003
• Article is provided courtesy of Prentice Hall Professional

Like this article? We recommend
Securing Systems with the Solaris Toolkit

TCP

The Transmission Control Protocol (TCP) provides connection-based, reliable data transport. It uses a lower protocol, IP, to manage the delivery of datagrams. TCP handles connection management and reliable data delivery. The network configuration options described here are managed in the Solaris OE TCP driver.

SYN Flood Attacks

In 1996, Issue 48 of the electronic journal Phrack contained an article, "Project Neptune," describing a network denial of service attack against TCP called SYN flooding. This attack makes a system respond very slowly (or not at all) to incoming network connections. A web site can appear to be down because it cannot establish connections for incoming browser requests. The Phrack article also contained source code to a program for initiating SYN flood attacks against remote systems. Soon after publication, several large Internet Service Providers (ISP) and web sites were victims of these network attacks. Attackers launched attacks from their dial-up modem connections to the Internet, which brought down sites with much faster connections to the network. Often it was difficult to trace the attack back to the source.

TCP is part of the TCP/IP network protocol suite and is connection-oriented. Prior to exchanging data using TCP, two systems must create a connection. Connection establishment is a three-step process in TCP, often called the three-way handshake. During this handshake, destination port information is exchanged and the two systems synchronize sequence numbers. (The SYN name refers to this synchronization step.)

The handshake works in the following manner:

1. A client sends a TCP segment to a server with the SYN flag set in the header, an Initial Sequence Number (ISN), and port number.

2. The server returns a segment to the client with the SYN flag set, an acknowledgement (or ACK flag), the original ISN + 1, and its own ISN.

3. The client sends a segment with the ACK flag set and the server's ISN + 1.

A connection is now established and data can be exchanged starting with the agreed upon sequence number.

The sequence numbers are used to provide reliability to the TCP protocol. The sequence numbers are incremented and sent with each outgoing packet. This allows the remote system to put packets in the proper order. If a packet is missing from the sequence, it can be detected and retransmitted.

The SYN flood attack takes advantage of the following weakness in the TCP protocol handshake. When a server receives the first SYN segment, it sends a SYN/ACK segment to the client address listed in the SYN segment. However, if that client is unreachable, the server will resend the SYN/ACK segment until a time limit is reached. (ICMP errors returned by the IP layer are ignored by the TCP layer.) If an attacking host sends many SYN segments for unreachable hosts, the server spends much time and system resources attempting to establish connections. Eventually, the server will reach its maximum of partially open connections. These incoming connections still in the handshake phase are part of the backlog queue for the specified port. In older versions of Solaris OE, the backlog queue was small. Once the queue is full, no further incoming SYN segments can be processed. Either the system will no longer respond for the specified port or the initial response becomes very sluggish. Systems with many network services could exhaust system memory because of the high number of uncompleted connections in the backlog queues.

In response to this attack, the Solaris 2.5.1 OE kernel TCP connection queue was changed and patches were issued. Previously, the size of the connection queue defined the size of the backlog queue. Now, there are two queues. There is still the queue for established connections. The new queue is for unestablished connections, where the handshake process is incomplete. SYN flood attacks affect this queue. When an attack occurs and the unestablished connection queue fills, an algorithm drops the oldest SYN segments first and allows the legitimate connections to complete. Patch 103582-11 (and later) adds this new queue system to the Solaris 2.5.1 OE release. The Solaris 2.6, 7, 8, and 9 OE releases have it incorporated. When a system is under attack, this message will appear in the logs:

Mar 8 19:24:01 example unix: WARNING: High TCP connect timeout rate!
[ccc] System (port 80) may be under a SYN flood attack!

This message indicates that the system is handling the attack as designed.

The sizes of the new queues are adjustable. Busy web servers might need to increase the size of the unestablished connection queue. The default size of the queue is 1024. Use this ndd command to increase it to 4096:

# ndd -set /dev/tcp tcp_conn_req_max_q0 4096

Add this command to the system startup scripts, or use the script described in "Sample System nddconfig init Script" on page 26. Any time a kernel queue is increased in size, there must be adequate system memory to handle the increase.

Connection Exhaustion Attacks

While SYN flood attacks attack the TCP three-way handshake, connection exhaustion attacks work on established connections. These attacks are not common because the connections can be traced back to the source in most cases, unlike SYN flood attacks. Most operating systems have a limit on the number of established connections that can be maintained whether set by a kernel parameter or available physical memory. Once this limit is reached, no new connections are established. The active connections must be completed and closed before new connections are established. For most web servers, this limit is never reached due the fact that HTTP connections are typically short-lived. An attacker can open many connections to a server and hold them open for long periods of time, effectively pushing the server closer to its connection limit. A web server will close connections that have completed and accept new connections. An attacker who continually and quickly requests new connections will eventually hold all of the available connections. Normal users of the web server will receive messages indicating that the web server is not responding. This is another denial of service attack.

One defense against this type of attack can be provided by tuning kernel and application parameters. This is not a complete solution, since it is basically a battle of resources. Whoever has the most resources (systems, memory, etc.) will most likely win the battle. An attacker can spread the connection attacks out to multiple systems to increase the total connection requests. However, some application and kernel adjustments can be made to reduce the effectiveness of such attacks. Most web servers have a parameter that sets the connection timeout value. For example, the Apache 1.3.9 web server has a configuration parameter named Timeout (in /etc/apache/http.conf of the Solaris 8 OE) that sets the maximum time a connection can be established. Once this time limit is reached, the server closes the connection. Setting this value to a lower value shortens the timeout period. Additionally, the Solaris 2.5.1 (with patch 103582-11 or later), 2.6, 7, 8, and 9 OE releases have a common parameter to adjust the maximum number of established network connections. The default value is 128. Use this ndd command to increase the default value to 1024:

# ndd -set /dev/tcp tcp_conn_req_max_q 1024

Decreasing the connection time and increasing the maximum number of established connections should be sufficient to ride out most connection exhaustion attacks. It may still be possible to create an effective denial of service even with the changes. However, the attacker must devote significant resources to be successful.

IP Spoofing Attacks

Predictable ISNs make it possible for attackers to compromise some systems. The TCP three-way handshake discussed previously involves two systems synchronizing sequence numbers prior to data exchange. For each new connection, most systems use ISNs that have fixed and predictable counter increments. An attacker uses this knowledge to create a three-way handshake by predicting the required ISN to establish a connection and execute a command.

This is a sophisticated attack that involves exploiting a trust relationship between two systems. Typically, a remote shell command (rsh) is attempted due to the trust configuration of a .rhosts file. This attack is carried out with the attacker unable to see the packets returned from the target host. It is due to the fact that the attacker is not on the same local network and the packets will be destined for the spoofed host. For this example, assume host A trusts host B. An attacker on host C (on a different network) wants to execute a command on host A. The first step in this attack is to disable host B. This can be done using the SYN flood attack described earlier. The attacker then establishes a TCP connection (or several connections to judge network delays) to the target host to sample the ISN used. This will be used to predict the next ISN.

The attacker uses the following steps in the TCP three-way handshake:

1. The attacker creates a TCP segment with the SYN flag set and an arbitrary ISN. The source address is set to the trusted host, and it is sent to the target system.

2. The target system returns a segment to the trusted system with the SYN and ACK flags set, the attacker ISN + 1, and its own ISN. The attacker cannot see this packet.

3. The attacker waits a period of time to allow the SYN/ACK segment to be sent and then sends a segment with the ACK flag set and the predicted ISN + 1.

If the attacker predicts the target's ISN accurately, then the remote shell daemon (in.rshd) will believe it has a valid connection to the trusted host. The attacker can now execute a command on the remote system.

RFC 1948 defines a better method for generating ISNs to prevent IP spoofing attacks. Using the procedure defined in this RFC, each connection has a unique and seemingly random ISN. A system using this technique is now a difficult target for an attacker attempting to predict the ISN.

There are several settings available on Solaris OE systems: the predictable method (0), an improved method with random increment value (1), and the RFC 1948 method (2). The default method for all revisions of the Solaris OE is 1. The 2.6, 7, 8, and 9 releases have all of these methods. The Solaris 2.5.1 OE release only has methods 0 and 1. Solaris 2.6, 7, 8, and 9 OE releases should be modified to use method 2.

There are two mechanisms to implement this change. The first option is to edit the /etc/default/inetinit file and change this line:

TCP_STRONG_ISS=1

to

TCP_STRONG_ISS=2

Reboot the system after this change.

The second mechanism is to enable this method while a system is in operation. Use the following command:

# ndd -set /dev/tcp tcp_strong_iss 2

Unfortunately, the Solaris 2.5.1 OE software does not offer the RFC 1948 method, and there are no plans to backport it. There might be a minor performance penalty for using the RFC 1948 method.

TCP Reverse Source Routing

As previously discussed, source-routed packets define a specific routing path instead of allowing network routers to make such decisions. Systems should be configured to not forward source-routed packets even when IP forwarding is enabled.

Additionally, the Solaris OE can be configured to ignore the reverse route on incoming TCP source-routed packets. Normally, the reverse routing path is copied into all packets destined for the system from which they were received. With TCP reverse source routing disabled, source-routed packets are processed normally, except that the reverse route information is removed from all response packets. It is available in Solaris 8 OE and newer OS releases.

This feature is disabled by default and that configuration can be verified with the ndd command:

# ndd /dev/tcp tcp_rev_src_routes
0

Ignoring the reverse route prevents an attacker from spoofing another system during the TCP handshake process. It is also included in the init script in "Sample System nddconfig init Script" on page 26.