High-speed serial interconnects carry data from chip to chip, board to board, and server to server.
In a recent series, we looked at power distribution within an artificial-intelligence (AI) data center. Of equal importance are the high-speed interconnects that carry data among the chips, boards, and racks that populate a data center or smart factory. These interconnects have names such as PCIe, NVLink, CXL, Infinity Fabric, XGMI, UALink, USB4, and InfiniBand.
Q: Do these technologies compete with each other?
A: Some are vendor-specific, and your choice will be dictated by the supplier of your AI accelerator chips or other compute devices. In general, however, these technologies complement each other. Some are board-to-board interfaces, while others handle chip-to-chip or rack-to-rack communications.
Q: What about PCIe?
A: PCIe traces its history back to the original PCs. These PCs tended to have a lot of boards: a motherboard for the processor, and at the time you would also probably add an acoustic modem board to talk to the plain old telephone system (POTS), a printer board, a general-purpose I/O board for connecting mice and other peripherals, and, if you were lucky enough to have access to Ethernet, a network interface card (NIC). A shared parallel bus called the Industry Standard Architecture (ISA) connected all these boards. As PC technology advanced, the bus became wider and faster, becoming the Extended ISA (EISA) bus and finally the parallel Peripheral Component Interface (PCI).
However, it became clear that the parallel-bus structure faced severe data skew, crosstalk, and other problems that would hinder efforts to increase parallel data widths and speeds further.
Q: What was the alternative?
A: By the late 1990s, serializer/deserializer (SerDes) technology, which originated in long-distance telecommunications, had shrunk to the point where it could be used in PC environments, and the serial PCI Express (PCIe) 1.0 standard emerged in the early 2000s, offering a 2.5-GT/s per-lane transfer rate (Table 1). The latest version, PCIe 7.0, was released in 2025 with a per-lane transfer rate of 128 GT/s.[1] Over the years, the signaling method has changed as well. The first five generations used non-return-to-zero (NRZ) signaling, while versions 6 and 7 use four-level pulse amplitude modulation (PAM4). PCIe 8.0, planned for release in 2028 with a 256-GT/s transfer rate, is also expected to use PAM4.
| Version | Year | Signaling method | Per-lane Transfer Rate (GT/s) |
| 1.0 | 2003 | NRZ | 2.5 |
| 2.0 | 2007 | NRZ | 5.0 |
| 3.0 | 2010 | NRZ | 8.0 |
| 4.0 | 2017 | NRZ | 16.0 |
| 5.0 | 2019 | NRZ | 32.0 |
| 6.0 | 2022 | PAM4 | 64.0 |
| 7.0 | 2025 | PAM4 | 128.0 |
| 8.0 | 2028* | PAM4 | 256.0 |
*Expected release year
Table 1. PCIe evolution
Q: Where does PCIe fit into today’s computing environments?
A: In a modern data-center environment, PCIe links CPUs, AI accelerators, NICs, nonvolatile memory (NVM), solid-state drives (SSDs), and other peripherals, as shown in blue in Figure 1.
PCIe can also be instrumental in linking CPUs and accelerators to random-access memory, including high-bandwidth memory (HBM). The Compute Express Link (CXL) specification, which sits on top of the PCIe physical layer, adds a set of protocols that enable features such as coherence, which ensures that in a multiprocessor system, all processors with access to the same memory see the same data—that is, one processor is guaranteed to see the latest update that another processor may have made.
Q: What other interconnects will we find?
A: Zoom out, and you’ll likely see InfiniBand (blue in Figure 1) connecting multiple servers. InfiniBand has generally provided higher data rates and lower latencies than Ethernet. However, a version called RoCE, which stands for Remote Direct Memory Access over Converged Ethernet, is showing promise in the data-center environment. In June, the InfiniBand Trade Association,[2] which promulgates both the InfiniBand and RoCE standards, announced that of the TOP500.org’s 500 most powerful commercially available computer systems, InfiniBand connects 293, while RoCE connects 83, giving InfiniBand and RoCE a 75% combined share of the total.
Q: What if we zoom in?
A: If you zoom inside an individual CPU or accelerator chip, you may see a technology such as Infinity Fabric connecting multiple chiplets on a substrate. If you look at an AI accelerator board, you may see NVLink, UALink, or XGMI providing communications among multiple devices. We’ll look at these interfaces next time, and we’ll review PAM4 eye diagrams and describe some differences with PAM3. Then, we’ll look at the systems needed to test the components and boards, ranging from high-speed oscilloscopes to semiconductor automated test equipment (ATE).
References
[1] PCI-SIG Releases PCIe 7.0 Specification to Support the Bandwidth Demands of Artificial Intelligence at 128.0 GT/s Transfer Rates, PCI-SIG
[2] RDMA Networking Powers the June 2026 TOP500 while IBTA Advances New Specifications, InfiniBand Trade Association
Related EEWorld Online content
What is GPIB and is it obsolete?
What is PCIe gen 6 and how do I test it? Part 1
Choosing a test system for emerging PCIe 6.0 designs
How are NRZ and PAM different in an eye diagram?
What information does an eye diagram depict?
How PAM4 moves bits through networks