Home > Articles > Networking

This chapter is from the book

1.3 Network Troubleshooting Tools

At this point, you should be very familiar with the network topology separating the NFS client and server, and have verified that IP packets are taking the appropriate route through the network in both directions. If you have not yet performed these critical steps, refer to the earlier Section 1.1 "Analyze Network Layout" for instructions on how to collect this information. After running network throughput tests using the tools described in Section 1.2 "Measure Network Throughput Capabilities," if you believe your network is experiencing a performance throughput issue then it is time to troubleshoot the network itself.


Common Causes of Dropped Network Packets

In many cases, network throughput problems are caused by packets being dropped somewhere on the network, either by the NFS client or server system itself or at some intermediate point in the network separating the two systems. Some of the more common reasons network packets are dropped include:

  • Defective hardware (i.e. network interface cards, cables, switch ports, etc.)

  • Mismatching configuration settings between interface cards and switch equipment. The most common configuration issue is where one side of a connection is set to half-duplex and the other side to full-duplex, causing "late" collisions to be logged on the half-duplex side and FCS or CRC errors logged on the full-duplex side.7

  • Network interconnect device buffer memory exhaustion (described in Section 10.4)

  • UDP socket overflows occurring on the NFS server, indicating that not enough daemons are running to handle the inbound requests for a specific port

The goal of this phase of the investigation is to determine if the network throughput problem is affecting all IP traffic or only NFS. In some cases, the only tools that can detect these types of problems are external analyzers and reporting tools specific to your network hardware. HP does provide a number of software-based tools to help detect and analyze network problems. Two frequently used network troubleshooting tools are netstat(1) and lanadmin(1M).

1.3.1 netstat -s

The netstat(1) command can be used to display statistics for network interfaces and protocols, it can list active network connections, print routing tables, etc. When executed with the "-s" option, netstat returns a complete list of all network transport statistics (TCP, UDP, IP, ICMP, and IGMP) arranged by protocol.

A portion of "netstat -s" output is shown in Figure 1.9 and Figure 1.10. These two screen shots illustrate how this single command returns an enormous amount of information about the underlying network protocols, including the number of TCP packets sent and received, the number of UDP socket overflows and checksum failures that occurred, etc. Also readily available are statistics such as the total number of IP packets received, the ICMP port unreachable and source quench messages generated, etc. All of this information can be extremely useful when troubleshooting a network or protocol layer problem.

Figure 1.9Figure 1.9 netstat -s Output Showing TCP Statistics

Figure 1.10Figure 1.10 netstat -s Output Including UDP and IP Statistics

Using netstat and diff to troubleshoot a network problem

Listed below is an example of how to use the "netstat -s" command to help determine if packets are being lost somewhere in your network The underlined steps are the commands you type.

  1. Initialize the "before" file with the current date and time.

    # date > netstat.before
  2. Collect a baseline set of netstat -s statistics on both the NFS client and server and append the output to the "before" file created in step 1.

    # netstat -s >> netstat.before
  3. Perform a test that exhibits the performance problem using the TCP protocol (such as ttcp or netperf).

    # ttcp -stp9 -n 100000 server
  4. Initialize the "after" file with the current date and time.

    # date > netstat.after
  5. Collect a second set of netstat -s statistics on both the NFS client and server and append the output to the "after" file created in step 4.

    # netstat -s >> netstat.after
  6. Locate any differences between the "before" and "after" netstat outputs to identify which statistics were incrementing during the test.

    # diff netstat.before netstat.after

An example of the type of output returned by diff(1) from such an exercise is shown in Figure 1.11. Some of the TCP statistics to monitor are "data packets retransmitted," "completely duplicate packets," and "segments discarded for bad checksum." If these statistics are steadily increasing over time it would indicate that packet loss is occurring somewhere in the network or that a possible network hardware problem is causing TCP checksum failures.

Figure 1.11Figure 1.11 Comparing "before" and "after" netstat Outputs

Also of concern would be an increasing number of UDP "bad checksums," as this could indicate that an IP level device in the network (perhaps a router performing IP fragmentation) is not correctly fragmenting UDP datagrams as it forwards them. These datagrams would be discarded by the receiving system, forcing the sending system to re-send this data. UDP "socket overflows" usually indicates that an application, such as NFS, is receiving requests on a particular UDP socket faster than it can process them, and consequently discarding requests.8

Of the IP statistics reported, "fragments dropped" and "fragments dropped after timeout" would indicate packet loss is occurring in the network.

Using netstat and beforeafter to troubleshoot a network problem

Although the diff(1) procedure does work for locating differences in netstat outputs, interpreting diff output can be a bit cumbersome. Not only do you have to manually subtract the "after" numbers from the "before" numbers, since the diff output removes the subsystem header information lines, you need to carefully confirm which statistics you are interpreting. For example, netstat returns "# packets received" under the TCP heading and "# total packets received" under the IP heading. Confusing these two values could lead to incorrect conclusions about the health of your network.

To simplify this procedure of interpreting multiple sets of netstat output, HP developed a tool called beforeafter. This program takes two netstat -s output files and compares them against each other. The output from beforeafter looks identical to netstat -s output except that the statistics represent only the differences between the "before" and "after" files. The tool is available at: ftp://ftp.cup.hp.com/dist/networking/tools/beforeafter.tar.gz.

The procedure for using the beforeafter tool is the same as the diff method outlined earlier, with the exception of step 6. Instead of using the diff command to compare the files, the beforeafter tool should be used as follows:

# beforeafter netstat.before netstat.after

Figure 1.12 contains an example of the beforeafter output. Notice the output looks identical to that of "netstat -s"; however the statistics reported are not cumulative totals but instead represent the differences between the "before" and "after" files.

Figure 1.12Figure 1.12 beforeafter Comparing netstat -s Statistics

The beforeafter tool even calculates the difference in wall-clock times between the date contained in the "before" and "after" files. Looking at the date line in Figure 1.12 you can see that the "after" file was collected 2 minutes and 28 seconds after the "before" file. The test caused this system to send 600503 packets containing 819239972 bytes of data, and receive 57535 packets containing acknowledgement packets for 819239975 bytes.

Although the diff output and the beforeafter output reveal the same information, locating and quantifying the key statistics is much easier using the beforeafter tool.

1.3.2 netstat -p <protocol>

Once you have identified a subset of the netstat -s output that is of particular interest, you can limit the statistical output to a single protocol by using the "-p <protocol>" syntax. An example is shown in Figure 1.13. Confining the output to a single protocol can greatly simplify the process of identifying specific protocol-related problems compared to analyzing screens full of "netstat -s" output.

Figure 1.13Figure 1.13 netstat -p Output

1.3.3 netstat -r

As discussed earlier in the "Analyze Network Layout" section, it is critical to understand the path NFS packets take through the network as they move between the client and server. We saw how utilities such as traceroute and "ping -o" display the various hops taken by packets going between two network nodes. If the traceroute or "ping -o" output reveals that packets are not taking the route you expect them to, you should verify the routing tables on both the client and server to make sure they are correct. Ensuring the accuracy of the routing tables is especially important on systems with multiple network interfaces, where outbound packets potentially have several paths to their final destination.

On HP systems, the "netstat -r" command is used to display the routing tables. Figure 1.14 shows an example of this. In this example, the "-n" (do not resolve IP addresses to hostnames) and "-v" (verbose) options were used. Included in the output is the interface name associated with each IP address, as well as the PMTU (Path Maximum Transmission Unit) size for each interface. The MTU information can be very useful in environments where the NFS client and server are on different physical networks and packet fragmentation or translation needs to occur (for example FDDI to Ethernet).

Figure 1.14Figure 1.14 netstat -r Displaying Network Routing Tables

1.3.4 netstat -i

In large customer environments, particularly those where HP's MC/ServiceGuard9 product is used, it is not uncommon for NFS client and server systems to have multiple network paths to each other for redundancy reasons. In some cases the primary and backup interfaces are not equivalent in terms of their bandwidth capabilities. For example, the systems might use a Gigabit Ethernet interface as their primary connection and have a 100BT interface available as a backup connection. In these environments, sufficient care must be taken when configuring the NFS mount points to ensure that the traffic flows across the faster interface whenever possible. However, even with careful preparation, there is always a possibility that NFS traffic sent between the clients and servers will mistakenly use the slower interface.

A quick and easy way to verify which interface the majority of network traffic is using is to issue the "netstat -i" command and examine the inbound and outbound packet counts for all configured interfaces. Figure 1.15 provides an example of this output.

Figure 1.15Figure 1.15 netstat -in Output

By monitoring the inbound and outbound packet rates of the interfaces, you can quickly determine if an unusually high amount of network traffic is using what should be an "idle" or "backup" interface. If this appears to be happening, a network trace can be taken to determine the hostnames of the remote systems that are sending requests to the slower interface.

1.3.5 lanadmin(1M)

As stated earlier, dropped packets on the network can occur if there are problems with the network interface cards, cables, or connectors. While many hardware-based problems can only be detected and identified with external analyzers, HP-UX provides several software-based tools to help monitor the health of the interfaces. The commands available for checking the state of any specific interface card will vary based on interface type (i.e. FDDI, Gigabit Ethernet, etc.). However, the lanadmin(1M) utility applies to all network links and it should be queried first.

The lanadmin command allows a system administrator to display many useful statistics kept by the LAN driver subsystem, regardless of the interface type. Figure 1.16 shows a sample screen output returned by lanadmin.

Figure 1.16Figure 1.16 lanadmin Output

By reviewing this information you can learn a great deal about how the queried interface is configured and whether it has been logging any errors at the driver layer. For example, the output shown in Figure 1.16 indicates that this interface card is a 10/100BT card known to the system as device "lan0," the card is enabled and active, it is running at a speed of 100 Mbits/second with an MTU size of 1500, and it is currently configured to run in full-duplex mode. In some cases, this information alone can be enough to determine the cause of a network performance problem (i.e. in the case where LAN interfaces and network switch configurations don't match with regards to speed and duplex settings).

Also available in the lanadmin output are various error counts, collision rates, total inbound and outbound packet counts, etc. By monitoring these counters you can, with the assistance of an HP support representative, try to make a qualified determination as to whether a hardware problem exists somewhere in your network. When software-based analysis tools fail to identify the problem, external tools can be used to provide the definitive view of the traffic patterns on the network and to isolate a device that is losing packets.


The Importance of Patching LAN, Transport, and Network Drivers

HP continually strives to improve the quality of HP-UX by distributing software patches containing both defect fixes and functionality enhancements. Many of these fixes and enhancements can significantly improve the performance and behavior of critical system components, such as LAN Common, the Network Transports (TCP, UDP, IP), and the various Network Link Driver subsystems (100BT, 1000BT, FDDI, Token Ring, etc.).

Since NFS relies heavily upon the stability and performance of the network, it is strongly recommended that the latest LAN, Transport, and network link driver patches be installed on every HP-UX system in order to take advantage of these improvements. Contact HP support to obtain a current set of patches for your specific operating system. You can also generate a current patch list using the tools available at HP's IT Resource Center: http://itrc.hp.com.

For a detailed discussion on the importance of keeping your HP-UX NFS client and server systems patched with current code, refer to Appendix B "Patching Considerations."

InformIT Promotional Mailings & Special Offers

I would like to receive exclusive offers and hear about products from InformIT and its family of brands. I can unsubscribe at any time.


Pearson Education, Inc., 221 River Street, Hoboken, New Jersey 07030, (Pearson) presents this site to provide information about products and services that can be purchased through this site.

This privacy notice provides an overview of our commitment to privacy and describes how we collect, protect, use and share personal information collected through this site. Please note that other Pearson websites and online products and services have their own separate privacy policies.

Collection and Use of Information

To conduct business and deliver products and services, Pearson collects and uses personal information in several ways in connection with this site, including:

Questions and Inquiries

For inquiries and questions, we collect the inquiry or question, together with name, contact details (email address, phone number and mailing address) and any other additional information voluntarily submitted to us through a Contact Us form or an email. We use this information to address the inquiry and respond to the question.

Online Store

For orders and purchases placed through our online store on this site, we collect order details, name, institution name and address (if applicable), email address, phone number, shipping and billing addresses, credit/debit card information, shipping options and any instructions. We use this information to complete transactions, fulfill orders, communicate with individuals placing orders or visiting the online store, and for related purposes.


Pearson may offer opportunities to provide feedback or participate in surveys, including surveys evaluating Pearson products, services or sites. Participation is voluntary. Pearson collects information requested in the survey questions and uses the information to evaluate, support, maintain and improve products, services or sites, develop new products and services, conduct educational research and for other purposes specified in the survey.

Contests and Drawings

Occasionally, we may sponsor a contest or drawing. Participation is optional. Pearson collects name, contact information and other information specified on the entry form for the contest or drawing to conduct the contest or drawing. Pearson may collect additional personal information from the winners of a contest or drawing in order to award the prize and for tax reporting purposes, as required by law.


If you have elected to receive email newsletters or promotional mailings and special offers but want to unsubscribe, simply email information@informit.com.

Service Announcements

On rare occasions it is necessary to send out a strictly service related announcement. For instance, if our service is temporarily suspended for maintenance we might send users an email. Generally, users may not opt-out of these communications, though they can deactivate their account information. However, these communications are not promotional in nature.

Customer Service

We communicate with users on a regular basis to provide requested services and in regard to issues relating to their account we reply via email or phone in accordance with the users' wishes when a user submits their information through our Contact Us form.

Other Collection and Use of Information

Application and System Logs

Pearson automatically collects log data to help ensure the delivery, availability and security of this site. Log data may include technical information about how a user or visitor connected to this site, such as browser type, type of computer/device, operating system, internet service provider and IP address. We use this information for support purposes and to monitor the health of the site, identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents and appropriately scale computing resources.

Web Analytics

Pearson may use third party web trend analytical services, including Google Analytics, to collect visitor information, such as IP addresses, browser types, referring pages, pages visited and time spent on a particular site. While these analytical services collect and report information on an anonymous basis, they may use cookies to gather web trend information. The information gathered may enable Pearson (but not the third party web trend services) to link information with application and system log data. Pearson uses this information for system administration and to identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents, appropriately scale computing resources and otherwise support and deliver this site and its services.

Cookies and Related Technologies

This site uses cookies and similar technologies to personalize content, measure traffic patterns, control security, track use and access of information on this site, and provide interest-based messages and advertising. Users can manage and block the use of cookies through their browser. Disabling or blocking certain cookies may limit the functionality of this site.

Do Not Track

This site currently does not respond to Do Not Track signals.


Pearson uses appropriate physical, administrative and technical security measures to protect personal information from unauthorized access, use and disclosure.


This site is not directed to children under the age of 13.


Pearson may send or direct marketing communications to users, provided that

  • Pearson will not use personal information collected or processed as a K-12 school service provider for the purpose of directed or targeted advertising.
  • Such marketing is consistent with applicable law and Pearson's legal obligations.
  • Pearson will not knowingly direct or send marketing communications to an individual who has expressed a preference not to receive marketing.
  • Where required by applicable law, express or implied consent to marketing exists and has not been withdrawn.

Pearson may provide personal information to a third party service provider on a restricted basis to provide marketing solely on behalf of Pearson or an affiliate or customer for whom Pearson is a service provider. Marketing preferences may be changed at any time.

Correcting/Updating Personal Information

If a user's personally identifiable information changes (such as your postal address or email address), we provide a way to correct or update that user's personal data provided to us. This can be done on the Account page. If a user no longer desires our service and desires to delete his or her account, please contact us at customer-service@informit.com and we will process the deletion of a user's account.


Users can always make an informed choice as to whether they should proceed with certain services offered by InformIT. If you choose to remove yourself from our mailing list(s) simply visit the following page and uncheck any communication you no longer want to receive: www.informit.com/u.aspx.

Sale of Personal Information

Pearson does not rent or sell personal information in exchange for any payment of money.

While Pearson does not sell personal information, as defined in Nevada law, Nevada residents may email a request for no sale of their personal information to NevadaDesignatedRequest@pearson.com.

Supplemental Privacy Statement for California Residents

California residents should read our Supplemental privacy statement for California residents in conjunction with this Privacy Notice. The Supplemental privacy statement for California residents explains Pearson's commitment to comply with California law and applies to personal information of California residents collected in connection with this site and the Services.

Sharing and Disclosure

Pearson may disclose personal information, as follows:

  • As required by law.
  • With the consent of the individual (or their parent, if the individual is a minor)
  • In response to a subpoena, court order or legal process, to the extent permitted or required by law
  • To protect the security and safety of individuals, data, assets and systems, consistent with applicable law
  • In connection the sale, joint venture or other transfer of some or all of its company or assets, subject to the provisions of this Privacy Notice
  • To investigate or address actual or suspected fraud or other illegal activities
  • To exercise its legal rights, including enforcement of the Terms of Use for this site or another contract
  • To affiliated Pearson companies and other companies and organizations who perform work for Pearson and are obligated to protect the privacy of personal information consistent with this Privacy Notice
  • To a school, organization, company or government agency, where Pearson collects or processes the personal information in a school setting or on behalf of such organization, company or government agency.


This web site contains links to other sites. Please be aware that we are not responsible for the privacy practices of such other sites. We encourage our users to be aware when they leave our site and to read the privacy statements of each and every web site that collects Personal Information. This privacy statement applies solely to information collected by this web site.

Requests and Contact

Please contact us about this Privacy Notice or if you have any requests or questions relating to the privacy of your personal information.

Changes to this Privacy Notice

We may revise this Privacy Notice through an updated posting. We will identify the effective date of the revision in the posting. Often, updates are made to provide greater clarity or to comply with changes in regulatory requirements. If the updates involve material changes to the collection, protection, use or disclosure of Personal Information, Pearson will provide notice of the change through a conspicuous notice on this site or other appropriate way. Continued use of the site after the effective date of a posted revision evidences acceptance. Please contact us if you have questions or concerns about the Privacy Notice or any objection to any revisions.

Last Update: November 17, 2020