Home > Articles

Implementing and Managing Security for Network Communications

  • Print
  • + Share This
This chapter is from the book

Terms you'll need to understand:

  • Internet Protocol Security (IPSec)

  • Authentication

  • Authentication Header (AH)

  • Encapsulating Security Payload (ESP)

  • IPSec certificates

Techniques you'll need to master:

  • Planning and IPSec deployment

  • Configuring IPSec policies

  • Deploying and managing IPSec policies

You should always maintain secure communications within your network and between networks to whatever degree possible. Internet Protocol Security (IPSec) allows you to control security using flexible policies to adapt to any network. IPSec secures data in the private and public environments of your network by providing a strong cryptography-based defense.

In this chapter, we discuss securing your network with IPSec, which includes the following:

  • Planning an IPSec deployment

  • Configuring IPSec policies

  • Deploying and managing IPSec policies

Planning an IPSec Deployment

One of the best features of IPSec is its flexibility. Unfortunately, that's a double-edged sword when it comes to network design. IPSec can be configured for the needs of almost any network, but many decisions must be made regarding its configuration. Some decisions are made for you by the way in which you are using IPSec or by the types of client operating systems that you are running on your network; other decisions are made by you. We now discuss the steps that you must take to deploy IPSec, which include the following:

  • Deciding which IPSec mode to use

  • Planning authentication methods for IPSec

  • Securing authentication with IPSec

  • Testing the functionality of existing applications and services

Deciding Which IPSec Mode to Use

This decision is rather easy because it has already been made for you based on the way you are using IPSec. Basically, two different types of communications exist that you need to secure—communications within a network and communications between networks. You choose one of two modes of IPSec security—transport mode or tunnel mode—based on the type of communication you need to make secure.

Transport Mode

You can use IPSec in transport mode to secure communications between two computers on the same network. This can be server-to-server or server-to-client communications. Once configured, IPSec provides end-to-end security based on the authentication and encryption settings that you apply. Figure 3.1 illustrates the concept of IPSec transport mode.

Figure 3.1Figure 3.1 IPSec transport mode provides secure communication from endpoint to endpoint within a network.

Tunnel Mode

You can use IPSec in tunnel mode to secure communications between two networks. You define the endpoints of the tunnel, and the system maintains a secure connection between the endpoints. The endpoints are generally router interfaces. You can specify the types of encryption and authentication protocols that will be required for all traffic. Figure 3.2 illustrates the concept of IPSec tunnel mode.

Figure 3.2Figure 3.2  IPSec tunnel mode provides for secure communications between two networks.

Planning Authentication Methods for IPSec

Authentication is the process of a network entity proving its identity in a confidential manner. Because authentication must be confidential, computers need to be able to authenticate to each other using a secure method. IPSec uses three main methods of authentication. You make your choice based on the capabilities of your servers and clients and the requirements of your organization. The three main methods from which you can choose are as follows:

  • Kerberos

  • Certificates

  • Preshared key


Kerberos is the default authentication method for Windows 2000 Server and Windows Server 2003. With Kerberos, the client must prove its identity to the server and the server must also prove its identity to the client. This is referred to as mutual authentication. Kerberos can only be used with Windows 2000 Professional and later clients. You should use Kerberos when all of your clients are in the same domain or Kerberos realm and can authenticate using Kerberos and when you want to use a method that requires the least administrative effort.

Figure 3.3 illustrates that Kerberos is the default protocol in the IP security policy.


When an exam question specifies an authentication method that requires the least administrative effort, and all of the clients and servers are Windows 2000, Windows Server 2003, or Windows XP Professional, you should first determine whether all of the clients are in the same domain. If so, you should use the default Kerberos authentication method. If not, then other methods will have to be considered.


As we discussed earlier, certificates are a method of granting access to a user based on the user's unique identification and whether the user possesses the right keys or algorithms to "unlock" the appropriate doors. You issue a certificate to a computer as a means for it to hold the key and as a means for you to track that the computer has the key. You can stipulate in an IPSec policy, as shown in Figure 3.4, that the computer or user must use a certificate (Secure Server) or that it should use a certificate if it has one (Server).

Figure 3.3Figure 3.3 Kerberos is the default authentication protocol for Windows 2000 Server and for Windows Server 2003 IPSec.

Figure 3.4 illustrates the default policies that are included in Windows Server 2003. Microsoft recommends that you create your own policies by modifying the default polices or by starting from scratch and using the Create IP Security Rule Wizard. Depending on your settings, you will create a policy that must be met in its entirety for communications to continue or a policy that can be negotiated between computers.

Preshared Key

A preshared key is a string that you can use to authenticate computers as a last resort. You should not use a preshared key if any other authentication method is available. A preshared key is a symmetric key, which means that the same key that is used to encrypt can also be used to decrypt. You should configure the same string on all computers that you want to authenticate.

The main problem with a preshared key is that you have no way of knowing whether the key is discovered. Also, the key is not specific to any individual. Therefore, an attacker could use the key to authenticate to a network and you could not trace the attack back to an individual. In addition, the preshared key is stored in the Registry in plain text form. It can be configured from the Authentication Methods tab in the properties of an IPSec policy as shown in Figure 3.5.

Figure 3.4Figure 3.4 You can require or request certificates as part of an IPSec policy.

Figure 3.5Figure 3.5 You can use a string to configure a preshared key when no other authentication options exist.

Security Authentication with IPSec

As mentioned previously, your goal is to allow computers to authenticate to each other without anyone else being able to see the process. To facilitate this, the authentication process must be encrypted. Two main protocols can be used when authenticating with IPSec. These are Secure Hash Algorithm (SHA) and Message Digest 5 (MD5). In addition, after authentication takes place, you can use multiple levels of encryption for the data itself. You should be familiar with the following terms and settings for IPSec encryption. These settings are illustrated in Figure 3.6 and can be located by clicking the General tab of an IPSec policy, then the Settings button, and finally the Methods button.

  • Secure Hash Algorithm (SHA)

  • Message Digest 5 (MD5)

  • Data Encryption Standard (DES)

  • Triple DES (3DES)

Figure 3.6Figure 3.6 You can choose the type of encryption used by IPSec during authentication and during data transfer.

Secure Hash Algorithm (SHA)

Secure Hash Algorithm (SHA) is the accepted standard for securing authentication of computers working with government contracts. It is used as part of the Federal Information Processing Standard (FIPS). SHA is a very high-security method that uses a 160-bit encryption key.

Message Digest 5 (MD5)

Message Digest 5 (MD5) is used for most commercial applications. You can use MD5 to secure authentication as well as data. This high-security method uses a 128-bit encryption key. It also has a lower performance overhead than that of SHA.

Data Encryption Standard (DES)

Data Encryption Standard (DES) is the lowest encryption strength of the Diffie-Hellman algorithms. It produces only a 56-bit key and is therefore not recommended for use in a high-security environment.

Triple DES (3DES)

Triple DES (3DES) is a much stronger Diffie-Hellman algorithm than DES, and produces a 168– to 2048-bit key. It is recommended for use in medium- to high-security networks.

Testing the Functionality of Existing Applications and Services

Remember, the reason that you originally decided to use security methods was to protect the integrity and the productivity of the network. For this reason, you should always test the applications and services that you are running on your network to make sure that they can still function with the IPSec rules that you have configured. There are many reasons that an application could cease to function or function with errors. Most of these involve the fact that IPSec rules can be used to filter traffic. Sometimes these filters, as indicated in the following list, can cause unexpected results:

  • Filtered ports

  • Filtered IP addresses

  • Filtered protocols

You can use the IP Filter Wizard (see Figure 3.7) to create multiple filters in an IPSec policy rule. If you use more than one filter in a single IPSec policy rule, be aware that the order that the filters are processed in is not necessarily the order in which you are viewing them. Instead, the IPSec Policy Agent reads the policy, and the filters are processed into one ordered list that is sorted from the most to the least specific. You can use the IPSec Monitor console to view the filters sorted by their weight. If you change or delete a filter, the IPSec Policy Agent reorders the filters based on what remains. Because of this, you should always test applications that use IPSec after applying IPSec and after changing any filters. You should use a test lab, if one is available, to test the effect of IPSec rules before assigning them in a production environment.

Figure 3.7Figure 3.7 You can use the IP Filter Wizard to create multiple filters in an IPSec policy rule.


IPSec can now function through some Network Address Translation (NAT) infrastructures. As long as the version of NAT is compliant with the latest RFC and is configured to allow User Datagram Protocol (UDP) traffic, the Internet Key Exchange (IKE) protocol will detect the presence of NAT and use UDP-ESP encapsulation to allow the traffic to pass through.

  • + Share This
  • 🔖 Save To Your Account