Home > Articles > Security > Network Security

  • Print
  • + Share This
This chapter is from the book

Troubleshooting NAT

To troubleshoot NAT, you should first verify that each necessary step has been performed. This means:

  • Validate that an ARP entry exists for the translated IP (or that the translated IP is somehow being routed to the firewall).

  • Validate that a static host route exists on the firewall to route the translated IP address to either the untranslated address or the next hop address if the real system is more than one hop away from the firewall.

  • Validate anti-spoofing. Make sure that the destination IP address is being translated, and verify that the translated IP address will pass anti-spoofing checks.

  • Validate that the rules are set up correctly. Set any security policy rule that applies to a NAT host to track long, and ensure that address translation is happening as you expect.

Wherever a verification of the configuration fails, a packet sniffer can be your friend. The remainder of this section shows you what you should see in a packet sniffer, what you shouldn't, and how to fix it.

Although there are plenty of external packet-sniffing devices, they can be expensive and inconvenient to use. Fortunately, some operating systems come with their own. Solaris comes with a tool called snoop. IPSO, Linux, and AIX come with tcpdump. Both of these tools will be discussed briefly in this chapter. Windows NT/2000 machines come with a limited packet sniffer in Network Monitor, but you can obtain a free copy of Ethereal from http://www.ethereal.com, which works far better.

Since version 4.0, FireWall-1 has also come with its own packet-sniffing utility called fw monitor. Because it works at the same level as FireWall-1 (i.e., just after the MAC layer and before the network layer), its use in troubleshooting NAT issues is limited. fw monitor relies on INSPECT code and is discussed in Chapter 13.

Consider the network and configuration that was used in the earlier step-by-step example (see Figure 9-3). Let's assume that host 192.168.42.69 is attempting to connect to 192.168.0.13, the Intranet Web Server, which really resides at 10.0.10.80.

With a successful connection, a tcpdump of the external interface should show you the following (-i is what you use to specify an interface to listen to):

    # tcpdump -i eth-s1p1
    tcpdump: listening on eth-s1p1
   18:51:20.806020 arp who-has 192.168.0.13 tell 192.168.0.2
   18:51:20.806020 arp reply 192.168.0.13 is-at 0:11:22:33:44:55
   18:51:54.135062 192.168.42.69.1777 > 192.168.0.13.80: S
   1184222758:1184222758(0) win 16384 <mss 1460,nop,wscale
   0,nop,nop,timestamp[|tcp]> (DF) [tos 0x10]
   18:51:54.135062 192.168.0.13.80 > 192.168.42.69.1777: S
   1332216451:1332216451(0) ack 1184222759 win 32120 <mss
   1460,nop,nop,timestamp 2739310[|tcp]> (DF)
   18:51:54.415021 192.168.42.69.1777 > 192.168.0.13.80: . ack 1 win
   17376 <nop,nop,timestamp 11362405 2739310> (DF) [tos 0x10]

If you were to use snoop, you would see the following (-d on snoop allows you to specify an interface to listen to):

    # snoop -d hme0
    Using device /dev/hme (promiscuous mode)
    192.168.0.2 -> (broadcast) ARP C Who is 192.168.0.13 ?
    192.168.0.1 -> 192.168.0.2 ARP R 192.168.0.13 is 0:11:22:33:44:55
    192.168.42.69 -> 192.168.0.13 HTTP C port=1777
    192.168.0.13 -> 192.168.42.69 HTTP R port=1777
    192.168.42.69 -> 192.168.0.13 HTTP C port=1777

Note that you may not necessarily see the ARP packets, especially if the originator of the packet already has the MAC address in its ARP cache. If you see SYN, SYN/ACK, and ACK packets, the connection should be established.

ARPs

The first part of the communication you should see is the request for MAC addresses via an ARP packet. When everything is working correctly, you will see an exchange like the following on the external interface with tcpdump:

    18:13:20.806020 arp who-has 192.168.0.13 tell 192.168.0.2
    18:13:20.806020 arp reply 192.168.0.13 is-at 0:11:22:33:44:55

With snoop, it looks like this:

    192.168.0.2 -> (broadcast) ARP C Who is 192.168.0.13 ?
    192.168.0.1 -> 192.168.0.2 ARP R 192.168.0.13 is 0:11:22:33:44:55

If you do only see the first packet over and over again (e.g., the ARP who is), this means that nobody owns or is proving a proxy-ARP for the translated address. Add a proxy-ARP as described previously. In some cases (especially when Windows NT is the firewall), you may need to add a static host route on the external router.

SYN Packets with No Response

You should then see the SYN packet, which looks something like this with tcpdump:

     18:13:22.040132 192.168.42.69.1777 > 192.168.0.13.80: S 
     3911298019:3911298019(0) win 16384 <mss 1460,nop,wscale 
     0,nop,nop,timestamp[|tcp]> (DF) [tos 0x10]

With snoop, it looks like this:

    192.168.42.69 -> 192.168.0.13 HTTP C port=1777

If this packet repeats over and over again, one of four things may be wrong:

  • The security policy is dropping the packet. Check your logs for drops.

  • The packet is being sent to the wrong MAC address.

  • The packet is not being routed properly.

  • The packet isn't actually getting translated, thus it is getting ignored.

Verify that the MAC address it is being sent to is correct (in this case, it should be 0:11:22:33:44:55). In tcpdump, you do this with the –e flag, which adds the MAC address to the output. In snoop, the only way to do this is with the –v flag, which unfortunately is extremely verbose.

Also in this example, you are only going to show packets coming from host 192.168.0.13 by means of adding host 192.168.0.13 to the end of your tcpdump or snoop command line. This will only show packets going to or from 192.168.0.13.

    # tcpdump -e -i eth-s1p1 host 192.168.0.13
    tcpdump: listening on eth-s1p1
   18:21:49.201680 0:aa:bb:cc:dd:ee 0:55:44:33:22:11 ip 82:
   192.168.42.69.2000 > 192.168.0.13.80: S 90360382:90360382(0) win
   16384 <mss 1460,nop,wscale 0,nop,nop,timestamp[|tcp]> (DF) [tos 
    0x10]
   18:21:54.240965 0:aa:bb:cc:dd:ee 0:55:44:33:22:11 ip 82:
   192.168.42.69.2000 > 192.168.0.13.80: S 90360382:90360382(0) win
   16384 <mss 1460,nop,wscale 0,nop,nop,timestamp[|tcp]> (DF) [tos
    0x10]
   18:22:07.209125 0:aa:bb:cc:dd:ee 0:55:44:33:22:11 ip 82:
   192.168.42.69.2000 > 192.168.0.13.80: S 90360382:90360382(0) win
   16384 <mss 1460,nop,wscale 0,nop,nop,timestamp[|tcp]> (DF) [tos
    0x10]
    # snoop –v –d hme0 host 192.168.0.13
    Using device /dev/hme (promiscuous mode)
    ETHER: ——- Ether Header ——-
    ETHER: 
    ETHER: Packet 27 arrived at 16:47:50.83
    ETHER: Packet size = 58 bytes
    ETHER: Destination = 0:55:44:33:22:11,
    ETHER: Source   = 0:aa:bb:cc:dd:ee,
    ETHER: Ethertype = 0800 (IP)
    ETHER: 
    IP:  ——- IP Header ——-
    IP: 
    IP:  Version = 4
    IP:  Header length = 20 bytes
    IP:  Type of service = 0x00
    IP:     xxx. .... = 0 (precedence)
    IP:     ...0 .... = normal delay
    IP:     .... 0... = normal throughput
    IP:     .... .0.. = normal reliability
    IP:  Total length = 44 bytes
    IP:  Identification = 47535
    IP:  Flags = 0x4
    IP:     .1.. .... = do not fragment
    IP:     ..0. .... = last fragment
    IP:  Fragment offset = 0 bytes
    IP:  Time to live = 245 seconds/hops
    IP:  Protocol = 6 (TCP)
    IP:  Header checksum = f23f
    IP:  Source address = 192.168.42.69
    IP:  Destination address = 192.168.0.13
    IP:  No options
    IP: 
    TCP: ——- TCP Header ——-
    TCP: 
    TCP: Source port = 2000
    TCP: Destination port = 80 (HTTP)
    TCP: Sequence number = 90360382
    TCP: Acknowledgement number = 0
    TCP: Data offset = 24 bytes
    TCP: Flags = 0x02
    TCP:    ..0. .... = No urgent pointer
    TCP:    ...0 .... = No acknowledgement
    TCP:    .... 0... = No push
    TCP:    .... .0.. = No reset
    TCP:    .... ..1. = Syn
    TCP:    .... ...0 = No Fin
    TCP: Window = 8760
    TCP: Checksum = 0xda2b
    TCP: Urgent pointer = 0
    TCP: Options: (4 bytes)
    TCP:  - Maximum segment size = 1460 bytes
    TCP:

In the preceding case, MAC 0:aa:bb:cc:dd:ee (which is the MAC of the external router) is trying to send to MAC 0:55:44:33:22:11, which is not the correct MAC. This problem can usually be resolved by flushing the ARP cache on the external router and retrying the test.

If the packet is not being routed properly, you could see a reset (RST) packet (see the section "SYN Followed by RST"), or you could see an ICMP Destination Unreachable packet. Verify that the static host route for 192.168.0.13 is pointing to the next hop address (10.0.0.2 as show in Figure 9-3).

If the packet is not actually being translated, you will see it very clearly in a tcpdump or snoop on the internal interface:

    # tcpdump -i eth-s1p2 host 192.168.42.69
    tcpdump: listening on eth-s1p2
   18:13:22.040132 192.168.42.69.1777 > 192.168.0.13.80: S 
   3911298019:3911298019(0) win 16384 <mss 1460,nop,wscale 
   0,nop,nop,timestamp[|tcp]> (DF) [tos 0x10]
   18:18:25.040168 192.168.42.69.1777 > 192.168.0.13.80: S 
   3911298019:3911298019(0) win 16384 <mss 1460,nop,wscale 
   0,nop,nop,timestamp[|tcp]> (DF) [tos 0x10]
   18:40:30.040342 192.168.42.69.1777 > 192.168.0.13.80: S
   3911298019:3911298019(0)  win 16384 <mss 1460,nop,wscale
   0,nop,nop,timestamp[|tcp]> (DF) [tos 0x10]
    # snoop -d hme0
    Using device /dev/hme (promiscuous mode)
    192.168.42.69 -> 192.168.0.13 HTTP C port=1777
    192.168.42.69 -> 192.168.0.13 HTTP C port=1777
    192.168.42.69 -> 192.168.0.13 HTTP C port=1777

Normally, you should see the translated IP address on the internal interface. If you do not see translated packets, check your NAT rules.

SYN Followed by RST

If the packet that follows the SYN is an RST packet (with snoop, you need –V to see the TCP flags):

    # tcpdump -i eth-s1p1 host 192.168.0.13
    tcpdump: listening on le0
    18:13:22.040132 192.168.42.69.1777 > 192.168.0.13.80: S 
    3911298019:3911298019(0) win 16384 <mss1460,nop,wscale 
    0,nop,nop,timestamp[|tcp]> (DF) [tos 0x10]
    18:13:22.040132 192.168.0.13.80 > 192.168.42.69.1777: R 0:0(0) ack 
    3911298020 win 0 [tos 0x10]
    # snoop –V -d hme0
    Using device /dev/hme (promiscuous mode)
    ________________________________
    192.168.42.69 -> 192.168.0.13 ETHER Type=0800 (IP), size = 58 
    bytes
    192.168.42.69 -> 192.168.0.13 IP D=192.168.0.13 S=192.168.42.69 
    LEN=44, ID=47247
    192.168.42.69 -> 192.168.0.13 TCP D=80 S=1777 Syn Seq=3052932309 
    Len=0 Win=8760 Options=<mss 1460>
    192.168.42.69 -> 192.168.0.13 HTTP C port=1777
    ________________________________
    192.168.0.13 -> 192.168.42.69 ETHER Type=0800 (IP), size = 60 
    bytes
    192.168.0.13 -> 192.168.42.69 IP D=192.168.42.69 S=192.168.0.13 
    LEN=40, ID=61295
    192.168.0.13 -> 192.168.42.69 TCP D=64836 S=80 Rst Ack=3052932310 
    Win=0
    192.168.0.13 -> 192.168.42.69 HTTP R port=1777

then one of three things is wrong:

  • The firewall is rejecting the packet on anti-spoofing. Verify that 192.168.0.13 is permitted by your anti-spoofing configuration on the internal interface.

  • The remote server is not running the service specified (in this case, port 80, HTTP).

  • The packet is being routed incorrectly.

If the remote server isn't actually running the service, you will see the following in a tcpdump and snoop on the internal interface:

    # tcpdump -i eth-s1p1 host 10.0.10.80
    tcpdump: listening on le0
    18:13:22.040132 192.168.42.69.1777 > 10.0.10.80.80: S 
    3911298019:3911298019(0) win 16384 <mss 1460,nop,wscale 
    0,nop,nop,timestamp[|tcp]> (DF) [tos 0x10]
    18:13:22.040132 10.0.10.80.80 > 192.168.42.69.1777: R 0:0(0) ack 
    3911298020 win 0 [tos 0x10]
    # snoop –V -d hme0
    Using device /dev/hme (promiscuous mode)
    ________________________________
    192.168.42.69 -> 10.0.10.80 ETHER Type=0800 (IP), size = 58 bytes
    192.168.42.69 -> 10.0.10.80 IP D=10.0.10.80 S=192.168.42.69 
    LEN=44, ID=47247
    192.168.42.69 -> 10.0.10.80 TCP D=80 
    S=1777 Syn Seq=3052932309 Len=0 Win=8760 Options=<mss 1460>
    192.168.42.69 -> 10.0.10.80 HTTP C port=1777
    ________________________________
    192.168.0.13 -> 192.168.42.69 ETHER Type=0800 (IP), size = 60 bytes
    192.168.0.13 -> 192.168.42.69 IP D=192.168.42.69 S=10.0.10.80 LEN=40, ID=61295
    192.168.0.13 -> 192.168.42.69 TCP D=64836 S=80 Rst Ack=3052932310 
    Win=0
    192.168.0.13 -> 192.168.42.69 HTTP R port=1777

The internal interface should see the untranslated packets (10.0.10.80 is the system's real IP). If the packet is being routed to the wrong interface, you will also see the same behavior as in the preceding output, but you will see a reject on Rule 0 in the Log Viewer as well. Verify that the static host route is set up correctly.

Useful tcpdump Flags

Table 9-4 contains a list of some useful flags for tcpdump, which takes commands in the following format:

    tcpdump –i interface-name [other-flags] [expression]

Table 9-4 tcpdump Flags

Flag

Description

-e

Displays MAC addresses with each packet.

-i interface

Required. Specify an interface to listen on.

-l

Buffer stdout, which is useful for piping tcpdump output to other programs.

-n

Disables name resolution on packets shown.

-p

Do not put interface in promiscuous mode (i.e., only show packets destined for the host).

-r filename

Read tcpdump capture from specified file.

-s N

Capture N number of bytes for each packet. The default is 68. (Useful with –x or –X.)

-S

Print absolute TCP sequence numbers instead of relative ones.

-w filename

Write captured packets to specified file.

-x

Hex dump of received packets.

-X

Hex and ASCII dump of received packets (IPSO only).


tcpdump Expressions

All tcpdump commands can be followed by an expression that filters the displayed (or saved) packets so that only the packets that are interesting are shown. Some useful expressions are shown in Table 9-5.

Table 9-5 tcpdump Expressions

Expression

Description

port 80

Show all packets with source or destination port 80 (TCP or UDP).

host 192.168.0.3

Show all packets coming from or going to host 192.168.0.3.

host 192.168.0.3 and tcp port 80

Show all packets coming from or going to host 192.168.0.3 and that are TCP packets with a source or destination port of 80.

proto vrrp

Show all VRRP packets. On non-IPSO platforms, use "ip proto 112."

icmp

Show all ICMP packets.

\(src host 192.168.0.1 or src host 192.168.0.2\) and proto vrrp

Show all VRRP packets that originate from 192.168.0.1 and 192.168.0.2 (the \ before the parenthesis is to escape it for the shell).

ether host aa:bb: cc:dd:ee:ff

Show all packets that come from or go to the specified MAC address.

ip proto 50

Show all packets of IP Proto 50 (IPSec AH in this example).

ip[2:2] > 576

Show IP packets that are longer than 576 bytes ([2:2] refers to the specific byte location in the TCP header and its length).

tcp[13] & 0x12 != 0

Show only TCP SYN/ACK packets. tcp[13] refers to the 13th byte in the TCP header of the packet.

icmp[0]

Show ICMP type 0 packets (i.e., echo reply). Icmp[0] refers to the 0th byte in the ICMP header.

icmp[0] = 3 and icmp[1] = 4

Show ICMP type 3, code 4 packets. These happen to be a response to receiving a packet that is too large to process and has the Don't Fragment bit set.


Useful snoop Flags

Table 9-6 contains a list of some useful flags for snoop, which takes commands in the following format:

    snoop [flags] [expression]

snoop Expressions

All snoop commands can be followed by an expression that filters the displayed (or saved) packets so that only the packets that are interesting are shown.

Table 9-6 snoop Flags

Flag

Description

-d interface

Specify an interface to listen on.

-v

Verbose mode. All packet data is displayed for each packet.

-V

A less-verbose verbose mode. A summary line is displayed for each layer in the ISO model.

-n

Disables name resolution on packets shown.

-P

Do not put interface in promiscuous mode, which means only show packets actually destined for this host (not useful on a switched segment).

-i filename

Read packets previously captured from file filename.

-o filename

Write packets to capture file named filename.

-s numbytes

Capture numbytes bytes for each packet. Normally, all bytes in the packet are captured.

-p x,y

Show packets numbered between x and y. The first packet captured is 1.

-t [a|d|r]

Timestamp format: a (absolute, i.e., wall clock time), d (delta, since capture was started), and r (relative time).


Some useful expressions are shown in Table 9-7. Note that many of the expressions are similar to tcpdump.

Table 9-7 snoop Expressions

Expression

Description

port 80

Show all packets with source or destination port 80 (TCP or UDP).

host 192.168.0.3

Show all packets coming from or going to host 192.168.0.3. You can also omit the "host" qualifier as well.

host 192.168.0.3

Show all packets coming from or going to host 192.168.0.3 and that are

and tcp port 80

TCP packets with a source or destination port of 80.

Icmp

Show all ICMP packets.

from 192.168.0.1

Show all packets that originate from 192.168.0.1 or are destined for

or to 192.168.0.2

192.168.0.2.

ether aa:bb:

Show all packets that come from or go to the specified MAC address.

cc:dd:ee:ff

 

ip proto 50

Show all packets of IP Proto 50 (IPSec AH in this example).

greater 576

Show packets longer than 576 bytes. You can use the word "less" instead of "greater" to show packets smaller than 576 bytes.

tcp[13] &

Show only TCP SYN/ACK packets. tcp[13] refers to the 13th byte in the

0x12 != 0

TCP header of the packet.


  • + Share This
  • 🔖 Save To Your Account