DoS Attacks! How to Protect Your Windows 2003 Server

• By Jesse Smith
• Mar 4, 2005

Like this article? We recommend
Building High Availability Windows Server™ 2003 Solutions

For any business connected to the Internet, numerous threats can menace the stability of its servers. And in the last few years, we've been given a whole new spectrum of things to defend against.

One of the most damaging threats—the denial of service (DoS) attack—has become far too familiar to most administrators. They seem to be the attack of choice for malicious users for some time now because of their capability to take down not only the target server, but the entire network segment on which the server is operating. In this article, I'll show you how to make your Windows 2003 servers more secure by adding values to the registry.

Two of the more popular (if that's the word) DoS attacks are SQL Slammer and NIMDA, both of which I've had the pleasure of experiencing firsthand. Firewalls against DoS attacks are usually helpless: The ports are already open because they're needed by some service, such as a database server operating over the Internet.

A DoS attack works by flooding the target server with TCP/IP packet connection requests. They use fraudulent packets (packets that are spoofed with a random IP address), which flood the server with connections it can't complete. New connections are also denied, and the network becomes saturated with streaming packets that can't be resolved.

The simplest first line of defense is to simply close the TCP port that the DoS attack targeted, but this means service downtime that can cost your business a considerable amount of money. With both Slammer and NIMDA attacks, the threat was new, so businesses had to wait for a patch solution from Microsoft before their Web and database services were running again. Virus-protection software was also useless against these attacks.

Based on these two examples alone, you can see the importance of "hardening" your server against DoS attacks. One effective way to do this is to add some TCP/IP protection values to the registry. Doing so will help you guard against DoS attacks by having the server better detect whether an attack is occurring. It will then time-out and discard these incomplete connections much more quickly.

Securing Windows Server 2003 Against DoS Attacks with the Registry

Although adding protection values to the registry will help make your server less vulnerable to these attacks, keep in mind that these techniques will not completely eliminate the possibility of getting a DoS attack. To begin, we add these new values to the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\parameters key in the Win2k3 registry by using the Registry Editor (see Figure 1).

Figure 1 Viewing the default TCP/IP parameter values.

In Figure 1 you can see the default values for the TCP/IP parameters key. The first value we create will time-out the connection responses during a three-way TCP/IP handshake between our server and the source machine. For example, the source machine will send the packet with a synchronize request (SYN). This packet contains a set of flag bits that initiate the connection with the server. Our server responds with an SYN/ACK packet to the source machine. It is a packet that needs to initiate a connection back with the source machine (the SYN part of the packet), and then it needs to send an acknowledgement to the source that the request for a connection was accepted. The source machine sends back an ACK packet to make the handshake a successful round trip connection.

Because a typical DoS attack uses a random IP as the source IP, it is likely that the source IP or machine (if it exists) will send a reset packet (RST /ACK) packet back to the server, saying it did not make the connection request. What happens even more often is that the IP address will not exist (because it is a random number); the server will keep trying to initiate a connection by resending SYN/ACK and then RST/ACK (because it didn't get any ACK back) packets back to the bogus source IP address.

All this creates incomplete or half-open connections, which is why we need to time-out these connections by creating a value called SynAttackProtect.

To create the SynAttackProtect value, right-click in the right-hand pane of the Registry Editor and select New > DWORD Value. You should see the same default value name assigned to this value called New Value #1, as shown in Figure 2.

Figure 2 Creating the SynAttackProtect TCP/IP parameter value.

Rename the value SynAttackProtect. Double-click the name to bring up the Edit DWORD Value dialog box shown in Figure 3.

Figure 3 Viewing the Edit DWORD Value dialog box.

Using this dialog box, we can add any one of three understood data values (0, 1, or 2). The default is 0, which protects against typical DoS attacks (one computer attacking another). For more advanced protection—for example, against distributed DoS attacks (attacks from multiple computers at once)—use a 1 or 2 in the value data field. These two data values (1 and 2) use advanced values for additional control. If you want better protection, create the following advanced values with their default data values:

• TcpMaxHalfOpen—Default data value is 100
• TcpMaxHalfOpenRetried—Default data value is 80
• TcpMaxPortsExhausted—Default data value is 5
• TcpMaxConnectResponseRetransmissions—Default data value is 3

A data value of 1 for the SynAttackProtect value uses the values above to better determine whether a DoS attack is taking place and adjust the retransmission of SYN/ACK packets and connection time-outs according to the data values for the advanced values. A 2 value gives you the most protection; I recommend it because it times-out connections most quickly and it gives additional delays to other connections so existing ones can be dealt with more efficiently.