AI training and inference clusters are built from thousands of accelerators working together, and connecting them requires two architecturally distinct approaches operating at different scales: scale-up, connecting accelerators tightly within a single server or rack, and scale-out, connecting many such racks together into a much larger cluster. Understanding the difference, and why modern AI infrastructure needs both simultaneously, is foundational to understanding how an AI data center’s networking is actually built.
Scale-Up: Tight Coupling Within a Node or Rack
Scale-up interconnect connects accelerators, GPUs or other AI chips, within the same server or the same rack, at extremely high bandwidth and extremely low latency, so that multiple chips can function as close to a single logical unit as possible. Nvidia‘s NVLink and NVSwitch are the dominant examples: NVLink provides direct, high-bandwidth GPU-to-GPU connections, while NVSwitch extends that connectivity to allow many GPUs within a rack to communicate with each other at near-uniform bandwidth, rather than some GPU pairs having faster paths than others. This tight coupling matters most for workloads that need to split a single large computation, such as training a large language model, across many GPUs simultaneously, where the accelerators need to exchange intermediate results constantly and any added latency between them directly slows the whole job down.
Scale-Out: Extending the Cluster Beyond a Single Rack
Scale-out interconnect connects multiple racks or pods together into a much larger cluster, using network fabric rather than the direct chip-to-chip links scale-up relies on. The two dominant technology choices at this layer are InfiniBand, long the default for high-performance computing and AI clusters due to its low latency and native support for remote direct memory access, and Ethernet using RDMA over Converged Ethernet, which has closed much of the historical performance gap with InfiniBand while offering better compatibility with the broader networking ecosystem operators and enterprises already run. Scale-out bandwidth per link is typically lower than scale-up bandwidth within a rack, which is an intentional design trade-off: the workload distributed across a scale-out fabric usually needs less frequent, larger-grain communication between racks than the constant fine-grained exchange happening between GPUs within a single rack’s scale-up fabric.
| Dimension | Scale-Up | Scale-Out |
| Connects | Accelerators within a server or rack | Racks or pods across a larger cluster |
| Typical technology | NVLink, NVSwitch | InfiniBand, or Ethernet with RoCE |
| Latency priority | Extremely low, near-uniform across all pairs | Low, but tolerant of somewhat more variability |
| Distance | Centimetres to a few metres | Metres to the full data center floor |
| Primary use | Splitting a single large computation across many chips | Coordinating many such groups into a larger training or inference job |
Why Modern AI Clusters Need Both Layers Together
Neither layer substitutes for the other. A cluster built entirely on scale-up-style tight coupling would be prohibitively expensive and physically impossible to extend beyond a relatively small number of accelerators, since the direct-link topologies that make scale-up so fast don’t scale efficiently to thousands of nodes. A cluster built entirely on scale-out-style network fabric, without scale-up’s tighter coupling inside each node, would leave enormous performance on the table for the tightly coupled portions of a large training job, where accelerators need to exchange data far more frequently and with far less tolerance for latency than the scale-out layer is designed to provide. The practical architecture nearly every large AI cluster uses today combines both: a tightly coupled scale-up domain within each server or rack, feeding into a scale-out fabric that connects many such domains into the full cluster, with the workload’s own communication pattern determining how the boundary between the two is drawn.
The Ethernet Versus InfiniBand Question at the Scale-Out Layer
The choice between InfiniBand and Ethernet at the scale-out layer remains one of the more actively contested architecture decisions in AI infrastructure planning. InfiniBand’s long-standing advantage has been purpose-built performance for exactly this kind of high-performance, low-latency, RDMA-native workload, with a mature ecosystem specifically tuned for AI and HPC clusters. Ethernet’s advantage is broader ecosystem compatibility, a much larger base of engineering talent already familiar with it, and continued performance improvements through RoCE and related standards that have narrowed the historical gap considerably. Neither choice is universally correct: an organisation building a dedicated, large-scale AI training cluster from scratch may still favour InfiniBand’s proven performance ceiling, while an organisation integrating AI infrastructure into a broader existing data center environment, where Ethernet already dominates every other layer, increasingly finds RoCE-based Ethernet a more operationally coherent choice even if it involves a modest performance trade-off relative to a pure InfiniBand build.
How This Architecture Choice Affects Overall Cluster Cost
The scale-up and scale-out split isn’t purely a technical decision, it has direct cost implications worth understanding when planning an AI infrastructure investment. Scale-up interconnect, NVLink and NVSwitch in particular, adds meaningful cost per accelerator but delivers a proportionally larger performance gain for tightly coupled workloads, which makes it worth the investment specifically for training jobs that depend on frequent inter-GPU communication. Scale-out fabric costs scale differently, driven more by the number of ports, the optical or copper interconnect linking them, and the switch silicon coordinating traffic across the cluster, with cost per unit of bandwidth generally falling as cluster scale grows, though not linearly, since larger scale-out fabrics also require more sophisticated topology design to avoid bottlenecks. An organisation sizing an AI cluster investment benefits from modelling scale-up and scale-out costs separately against the specific workload’s communication pattern, rather than treating total interconnect spend as a single line item, since the right balance between the two varies significantly depending on whether the dominant workload is large-scale training, which leans more heavily on scale-up, or high-volume inference serving, which often tolerates a more scale-out-heavy architecture.
Real-World Cluster Topologies Combine Both in Practice
Production AI clusters typically organise accelerators into pods, a set of servers connected via scale-up interconnect within a rack or small group of racks, with multiple pods then connected via scale-out fabric into the full cluster. This pod-based approach lets an operator scale a cluster incrementally, adding pods as capacity needs grow, while keeping the tightly coupled scale-up domain within each pod small enough to remain cost-effective and technically manageable. The specific pod size, how many accelerators sit within a single scale-up domain before the boundary shifts to scale-out, is itself a design choice that varies by vendor and by workload, and it’s one of the more consequential architecture decisions an AI infrastructure buyer makes, since it directly determines how much of the cluster’s total interconnect benefits from scale-up’s lower latency versus how much depends on the more cost-efficient but higher-latency scale-out fabric.
Explore more Digital Infrastructure analysis on TeckNexus — https://tecknexus.com/intelligence/digital-infrastructure-monthly-insights/.
Also check OTN Capacity & Right-Sizing Calculator and AI Data Center Interconnect (DCI) Bandwidth Planner
