Optics and Cable Management
- Scaling Optics for AI Clusters
- Challenges in Optical Innovation
- Packet Flow
- Transmission Modes
- Transceiver Types
This chapter is from the book
This chapter is from the book
This chapter is from the book
Scaling Optics for AI Clusters
In Chapter 3, “Network Design Considerations,” we discussed design concepts involved in building an AI/ML data center. Optics and cables are important components of any data center, so requirements of an AI/ML data center.
With AI/ML clusters, the ports on a server currently support 200 Gbps and 400 Gbps throughput and are moving toward 800 Gbps and 1.6 Tbps throughput. Nvidia is the top vendor for the GPU server supplies for AI/ML clusters. It has multiple generations of GPUs—Volta, Ampere, Hopper, and Blackwell. Nvidia currently has A100 (Ampere) and H100 (Hopper) on the market and is moving toward H200 and further versions of GPU based on Blackwell.
From the GPU, Nvidia uses NVLink to connect to the NVSwitch, which is used within the server to communicate between the GPUs that are internal to the server. Each NVLink is 300 Gbps for the Volta generation, 600 Gbps for the Ampere generation, and 900 Gbps for the Hopper generation, moving toward 1800 Gbps for the Blackwell generation. Other vendors have similar solutions; for example, AMD has Infinity Fabric, and Intel has CXL (Compute Express Link), UCIe (Universal Chiplet Interconnect Express), and PCIe (Peripheral Component Interconnect Express) switches.
Figure 4-1 illustrates the system topology of a server with 8 GPUs, where an internal switch helps with communication across the GPUs in the server.
Figure 4-1
System topology
In addition to the NVSwitch for internal communication, there are different NICs that are used for external connectivity between the GPUs across multiple servers. In either case of NVIDIA or AMD-based GPU servers for the scale-out DC use-case, each GPU is connected to a dedicated NIC card, and then each NIC card from the same server connects to different top-of-rack Ethernet or InfiniBand switches. It means for local intra-server communication, the NV switch will be used, and for any server-to-server communication, the external switch is used. GPU-to-NIC card connections are 400 Gbps or 800 Gbps Ethernet or IB. This means a top-of-rack switch is also typically a high-port-density 400Gbps/800Gbps and 1.6Tbps switch. From the AI DC server perspective, besides the GPU-connected NIC cards, there are also storage NIC cards (NVMe-o-F, for example) and out-of-band connections. They are all interconnected via PCIe generation 5 and newer to offer even higher local server interconnects to memory blocks or the CPU.
Figure 4-2 shows a chart from a Dell’Oro report on market adoption of optics for AI clusters from 2020 to 2027. This chart indicates that optics adoption is going to move toward 1.6 Tbps. With the requirement of high-bandwidth optics, there is also a need for high-radix switches that can support a large number of ports per rack unit (RU). In addition, optics need to be power-efficient to reduce the power and thermal budget of a rack. These needs are driving the enhancement of small-form-factor optics, modulation, connectors, and cables.
)
Figure 4-2.1 shows a 2025 study from Dell’Oro showcasing the adoption of 800 Gbps in 2025 and growing demand for 3.2 Tbps optics by 2029.
)
The throughput requirement from server to leaf is expanding to 200 Gbps/400 Gbps/800 Gbps/1.6 Tbps, which is leading vendors to focus on optics to support high-bandwidth connections.
){kind=link}