- Overview
- Categorizing TCP Traffic
- Gigabit Ethernet Latency on a Sun Fire 6800 Server
- Bulk Transfer Traffic Performance
- Small Packet TCP Traffic Performance
- Summary
- About the Author
- References
- Ordering Sun Documents
- Accessing Sun Documentation Online
Like this article? We recommend
Like this article? We recommend
Like this article? We recommend
Gigabit Ethernet Latency on a Sun Fire 6800 Server
As discussed in "Bulk Transfer Traffic Performance Issues," the latency determines the minimal size of the TCP window to achieve one Gbps for bulk transfer traffic. This latency also affects the user-perceived response time. This section discusses the latency for processing packets on the Sun Fire 6800 platform. The TCP Request-Response (TCP RR) tests were conducted using MC-Netperf [3], an extended version of the popular Netperf [2] tool.
The metric for the TCP RR test is transaction rate. A transaction includes these processes—client sending a packet to server (similar to a request), server processing the packet, server sending the client a packet of the same size (similar to a reply), and the client processing the packet. Both the client and the server deal with one request at a time. In this case, both the server and the client are domains of a Sun Fire 6800 server with identical hardware configuration. Both domains use the same driver for the network interface card and the same release (02/02) of the Solaris 8 OE. As a result, assume that the packet processing time is the same for both domains. Hence, a transaction includes two symmetric operations. Each operation includes sending a packet in software, sending a packet in hardware, receiving a packet in hardware, and receiving a packet in software. The latency of one such operation is defined as the latency of gigabit networking on Sun Fire servers. For this article the following data points were collected using one card and one TCP connection for a comprehensive evaluation:
Latency when the system is conducting bulk transfer. The payload size is 1,460 bytes.
Latency when the system is handling small packets. Payload size varied from four bytes to 1,024 bytes.
Since the highest transaction rate for the preceding data points indicates the minimum latency, the goal should be to tune the system until it delivers the highest transaction rate. As discussed in the previous section, the setting for deferred acknowledgment, that is, the value of tcp_deferred_acks_max2, may impact the transmission overhead, and hence may impact the transaction rate. However, the unit for tcp_deferred_acks_max is in TCP MSS3. If the tests only have messages no larger than 1,460 bytes, setting this parameter to one disables deferred acknowledgment, while setting this parameter to any value larger than one enables it. It is intuitive to think that Nagle's algorithm should also have an impact on the transaction rate. However, this algorithm does not govern the timing to send the first small packet, and the TCP RR test will only have one small packet outstanding at any time, thus the impact of Nagle's algorithm can be ignored in this case.
Different settings of Nagle's algorithm and deferred acknowledgment were tested. TABLE 1 shows the results.
Table 1 TCP RR Transaction Rates (One Card, Single Connection, Four CPUs)
|
Payload (bytes) |
Deferred ACK off |
Deferred ACK on |
|
4 |
6972 |
6978 |
|
40 |
6990 |
6885 |
|
100 |
6714 |
5358 |
|
250 |
5452 |
5634 |
|
536 |
4973 |
5157 |
|
1024 |
4196 |
4302 |
|
1460 |
3104 |
3244 |
The transaction rates peak at 6,978 transactions-per-second and the transaction rates for 4-byte, 40-byte, and 100-byte payloads are very close. Disabling deferred acknowledgments lowers transaction rates for most cases. Using the snoop utility shows that both parties in the communication lose the opportunity to piggy-back the ACK in this case. As a result, both parties must process one additional ACK packet for each transaction. Enabling deferred acknowledgment is preferred for this type of traffic. The minimum latency for processing any packets is 6,978 transactions-per-second, which is 140 microseconds round-trip and about 70 microseconds one-way. For packets with a payload of 1,460 bytes, the transaction rate stays about 3109 to 3244 per second. This is roughly 150 microseconds one way. Hence, the latency for gigabit Ethernet on the Sun Fire 6800 is between 70 and 150 microseconds.