InformIT: Red Hat Linux 7 Unleashed > TCP/IP Troubleshooting Tools

TCP/IP Troubleshooting Tools

Problems rarely appear once a TCP/IP network is configured. However, networking equipment fails, lines go down, and cables get disconnected. Also, problems can arise during the initial configuration of a networked host.

Linux has three basic network troubleshooting tools. Two of them, ping and traceroute, are concerned with the capability of a host to reach another, while the third one, tcpdump, is useful for analyzing the flow of traffic in a network.

`ping`

The most basic network troubleshooting tool is the ping program. Named after the pinging sound made by submarine sonars, ping sends out packets to another host and waits for that host to reply to them. ping uses ICMP (Internet Control Message Protocol), a protocol that runs over IP and is designed for control messages used for things such as routing and reachability information.

The most common way of using ping is to pass it a hostname or address:

% ping server.company.com
PING server.company.com (10.0.1.10): 56 data bytes
64 bytes from 10.0.1.10: icmp_seq=0 ttl=245 time=83.239 ms
64 bytes from 10.0.1.10: icmp_seq=1 ttl=245 time=80.808 ms
64 bytes from 10.0.1.10: icmp_seq=2 ttl=245 time=82.994 ms
64 bytes from 10.0.1.10: icmp_seq=3 ttl=245 time=81.635 ms
^C
--- server.company.com ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 80.808/82.169/83.239 ms

In this case, ping pings the target host one time per second, until you press Ctrl+C. At that moment, it prints out the statistics for the run. In the statistics, aside from the number of packets transmitted and received, you can see the minimum, average, and maximum round-trip times, which will help you find out how congested the path to the destination host is at the moment.

ping has many options. Here are the most commonly used:

Option	Function
`-c` `count`	Only sends `count` number of packets instead of pinging forever.
`-n`	`ping` displays numeric addresses instead of hostnames. Useful when you can't get to the DNS server or when DNS queries take too long.
`-r`	Record route. Sends an option in every packet that instructs every host between the source and the target to store its IP address in the packet. This way you can see which hosts a packet is going through. However, the packet size is limited to nine hosts. Some systems disregard this option. Because of this, it is better to use `traceroute`.
`-q`	Quiet output. Outputs just the final statistics.
`-v`	Verbose. Displays all packets received, not just `ping` responses.

When troubleshooting network problems, you should first ping the IP address of the source host itself. This verifies that the originating network interface is set up correctly. After that, you should try pinging your default gateway, your default gateway's gateway (the next hop out), and so on, until you reach the destination host. That way you can easily isolate where a problem lies. However, once you've verified that you can get to the default gateway, it is better to use the traceroute program (which is described in the following section) to automate the process.

NOTE

All TCP/IP packets have a field called Time-To-Live, or TTL. This field is decremented once by each router on the network. The packet is discarded the moment it reaches 0. Whereas ping uses a default TTL of 255 (the maximum value), many programs such as telnet and ftp use a smaller TTL (usually 30 or 60). That means that you might be able to ping a host, but not telnet or FTP into it. You may use the -t ttl option to ping to set the TTL of the packets it outputs.

`traceroute`

The traceroute program is the workhorse of TCP/IP troubleshooting. It sends out UDP packets with progressively larger TTLs and detects the ICMP responses sent by gateways when they drop the packets. In the end, this maps out the route a packet takes when going from the source host to the target host.

This is how it works: traceroute starts by sending out a packet with a TTL of 1. The packet gets to a gateway, which can be the target host or not. If it is the target host, the gateway sends a response packet. If it isn't the target host, the gateway decrements the TTL. Since the TTL is now 0, the gateway drops the packet and sends back a packet indicating this. Whatever happens, traceroute detects the reply packet. If it has reached the target host, its job is finished. If not (it received notification that the packet was dropped, for instance), it increments the TTL by 1 (its new value is 2) and sends out another packet. This time the first gateway decrements the TTL (to 1) and passes it through to the next gateway. This gateway does the same thing: determines whether it's the destination host and decrements the TTL. This goes on until either you reach the target host or you reach the maximum TTL value (which is 30 by default, but can be changed with the -m max_ttl option).

traceroute sends three packets with each TTL and reports the round-trip time taken by each packet. This is useful for detecting network bottlenecks.

traceroute is usually used the same way as ping--by giving it a destination address. Listing 20.1 shows an example of the output from traceroute.

Example 20.1. Sample Output from `traceroute`

mario@chaos:~ 511 $ /usr/sbin/traceroute www.umbral.com
traceroute to xmaya.umbral.com (207.87.18.30), 30 hops max, 40 byte packets
 1:  master.spin.com.mx (200.13.80.123)  120.75 ms  126.727 ms  109.533 ms
 2:  octopus.spin.com.mx (200.13.81.32)  110.042 ms  104.654 ms  99.599 ms
 3:  200.33.218.161 (200.33.218.161)  119.539 ms  105.697 ms  109.603 ms
 4:  rr1.mexmdf.avantel.net.mx (200.33.209.1)  131.556 ms  112.767 ms 109.6 ms
 5:  bordercore1-hssi0-0.Dallas.cw.net (166.48.77.249)  159.54 ms  155.378 ms 169.598 ms
 6:  core9.Dallas.cw.net (204.70.9.89)  159.483 ms  156.364 ms  159.628 ms
 7:  dfw2-core2-s1-0-0.atlas.digex.net (165.117.59.13)  169.505 ms 156.024 ms  149.628 ms
 8:  lax1-core1-s8-0-0.atlas.digex.net (165.117.50.25)  199.497 ms 194.006 ms  189.621 ms
 9:  sjc4-core2-s5-0-0.atlas.digex.net (165.117.53.74)  199.489 ms
    sjc4-core2-s5-1-0.atlas.digex.net (165.117.56.110) 191.025 ms 
    sjc4-core2-s5-0-0.atlas.digex.net (165.117.53.74) 210.25 ms          
10:  sjc4-core6-pos1-1.atlas.digex.net (165.117.59.69)  201.031 ms 196.195 ms  199.584 ms
11:  sjc4-wscore2-p1-0.wsmg.digex.net (199.125.178.37)  360.468 ms 366.267 ms  199.481 ms
12:  sjc4-wscore4-fa1-0.wsmg.digex.net (199.125.178.20)  582.272 ms 207.536 ms  198.275 ms
13:  xmaya.umbral.com (207.87.18.30)  209.457 ms  3076.14 ms *

traceroute can give you quite a bit of information if you know how to look for it. For example, you can see a few things in Listing 20.1:

http://www.umbral.com is actually an alias for xmaya.umbral.com. traceroute always does a reverse DNS lookup and reports the official hostname of the host it's tracing.
xmaya.umbral.com is connected to the Internet through a service provider whose domain is digex.net (probably an ISP called Digex). (Lines 7–12 are all in the IP networks belonging to the domain of the last gateway.)
You are connected to the Internet through an ISP called Spin, which is in Mexico. (Lines 1 and 2 are the first gateway. The domain ends with .mx.)
The digex hosts that appear on line 9 are actually several hosts with the same IP address. This is done for redundancy.
There seems to be some kind of bottleneck between hosts 10, 11, and 12. Notice how the response time, after slowly growing steadily until line 9, suddenly jumps from about 200 milliseconds to more than 300, and then to more than 500. This might be a temporary bottleneck (caused simply by the traffic load at the moment) or it may be a continuous problem, caused perhaps by a physical media problem or not enough capacity in the link.

As you can see, traceroute can be an invaluable tool. Much more information can be gleaned from traceroute output; it is best to read the traceroute(8) man page for a complete discussion.

`tcpdump`

tcpdump is another invaluable tool for debugging some types of network problems. It basically works as a packet sniffer--it listens to the network, looks at any packets that come by (whether destined for the host on which it is running or not), and operates on it. It can store all or just some interesting parts of the traffic it sees, or perform a rudimentary analysis of the information it contains.

tcpdump works by setting the network card into what is known as promiscuous mode. Normally, a network card will only see packets that are meant for it. However, in promiscuous mode, it will see all packets that pass through the network and pass them to the operating system above. The OS then passes the packets to tcpdump, which can then filter and display or store them. Because it modifies the configuration of the network card, tcpdump must be run by root.

CAUTION

tcpdump is a potential security hole. It falls into the category of programs known as sniffers, which listen to the network and can listen to all packets in the network and store them. If users use programs such as Telnet that send passwords in the clear, a cracker might use a sniffer to sniff out their passwords. Because of this, tcpdump should never be installed setuid-root.

To detect whether a network interface is in promiscuous mode (and thus might have a sniffer running on it), use the ifconfig command to display the interface's configuration. The PROMISC flag will appear if the interface is in promiscuous mode:


      # /sbin/ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:60:97:3E:F0:61 
          inet addr:10.0.1.50  Bcast:10.0.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5 errors:0 dropped:0 overruns:0 carrier:5
          Collisions:0
          Interrupt:3 Base address:0x310

If you run tcpdump without any arguments, you get a listing of all the packets that pass through the network:


   # /usr/sbin/tcpdump
tcpdump: listening on eth0
22:46:12.730048 renato.1445323871 > vishnu.nfs: 100 readlink [|nfs]
22:46:12.734224 tumbolia.1012 > vishnu.808: udp 92
22:46:12.746763 tumbolia.22 > atman.1023: P 142299991:142300035(44) ack 3799214339 win 32120 (DF) [tos 0x10]
22:46:12.763684 atman.1023 > tumbolia.22: . ack 44 win 32120 (DF) [tos 0x10]
22:46:12.778100 vishnu.808 > tumbolia.1015: udp 56
22:46:12.780084 gerardo.1448370113 > vishnu.nfs: 124 lookup [|nfs]
22:46:12.780153 tumbolia.22 > atman.1023: P 44:596(552) ack 1 win 32120 (DF) [tos 0x10]

The dump will stop when you press Ctrl+C.

As you can see, tcpdump by default converts IP addresses to hostnames and port numbers to service names. It also attempts to interpret some packets (such as those where the line ends with lookup [|nfs], which are NFS lookups). In some cases, the number of bytes that tcpdump looks at (68) might not be enough to fully decode the packet. In this case, you may use the -s option to increase the number (see the -s option in Table 20.5).

You don't often want to see all the packets, especially in medium to large networks. Sometimes you want to see all the packets going between two specific hosts, or even those that use a specific service. tcpdump takes as a parameter an optional filter expression that will select only certain packets.

tcpdump's filter expressions consist of one or more primitives joined by the keywords and, or, and not. Each primitive consists of a qualifier followed by an ID. A qualifier consists of one or more keywords, the most common of which are shown in Table 20.4. The ID specifies the value the corresponding field must have to match the filter.

Table 20.4. Most Common `tcpdump` Qualifiers

Qualifier	Matches
`src host`	The IP address of the host from where the packet comes.
`dst host`	The IP address of the host to which the packet is going.
`host`	The IP address of the source or the destination host.
`src port`	The port the packet is coming from.
`dst port`	The port the packet is going to.
`port`	The source or the destination port.
`tcp`, `udp`, or `icmp`	The packet's protocol is the specified one.

TIP

One common mistake is to run tcpdump through a remote connection, such as when connected through telnet or ssh, with a filter that includes all the telnet or ssh packets. An example is including just host thishost, where thishost is the host on which tcpdump is running. In that case you end up with an incredible amount of output. This is because the first packet that comes through generates output, which is transmitted through the network and captured by tcpdump, which generates more output, which is also transmitted through the network, and so on.

To prevent that, be more specific in your filter expressions. For example, you might include the primitive not port 22 to filter out ssh packets.

tcpdump also takes several switches, the most common of which are shown in Table 20.5.

Table 20.5. Most Common `tcpdump` Switches

Qualifier	Matches
`-c` `count`	Exit after receiving `count` packets.
`-i` `interface`	Listen on `interface`. By default, `tcpdump` listens on the first interface found after the loopback interface. You can see the order in which the interfaces are searched using the `ifconfig -a` command.
`-n`	Don't convert numeric addresses and port numbers to host and service names (print numeric output). This eliminates the need for DNS lookups.
`-N`	Print out only the hostname, not its fully qualified domain name. For example, if you give this flag then `tcpdump` will print redhat instead of redhat.com
`-r` `file`	Read packets from `file`, which must have been created with the `-w` option.
`-s` `snaplen`	Grab `snaplen` bytes from each packet. The default is 68, which is enough for IP, ICMP, TCP, and UDP packets. However, for certain protocols (such as DNS and NFS), a snaplen of 68 will truncate some protocol information. This is marked by `\|protocol`, where the `protocol` indicates the protocol part where truncation occurred.
`-v`	Verbose mode. Print some more information about each packet.
`-vv`	Even more verbose output. Print much more information about each packet.
`-w` `file`	Capture the packets in `file`.
`-x`	Print out each packet in hex. This will print out either the whole packet or snaplen bytes, whichever is less.