Scale-Up vs Scale-Out: What Bandwidth Does Your AI Cluster Actually Need Before the RFP Goes Out?

A four-phase planner separates in-cluster accelerator interconnect demand from cross-site DCI bandwidth need, calibrated by accelerator generation, workload mix, and distributed training pattern — with a latency feasibility check most AI infrastructure RFPs skip entirely

AI infrastructure network planning frequently conflates two genuinely different bandwidth problems: how fast accelerators inside a single cluster need to talk to each other, and how much bandwidth actually needs to cross between physical sites. Getting that distinction wrong in an RFP either massively over-specifies expensive inter-site connectivity that a workload never needed, or under-specifies it in a way that silently throttles distributed training performance across campuses. TeckNexus has published an AI Data Center Interconnect (DCI) Bandwidth Planner, a four-phase tool that models scale-up and scale-out bandwidth requirements separately, grounded in publicly published accelerator interconnect generations and well-documented distributed-training traffic patterns.

Accelerator generation sets per-unit interconnect bandwidth — and the range is enormous

The planner’s first input pairs accelerator count in the largest cluster with accelerator generation tier — prior-generation at roughly 350 GB/s per accelerator, current mainstream at roughly 750 GB/s, latest-generation at roughly 1,700 GB/s, and next-gen/frontier hardware at roughly 3,200 GB/s. That’s close to a tenfold range across four generation tiers, and it directly determines per-unit interconnect bandwidth demand — meaning the same accelerator count can imply radically different network requirements depending purely on which hardware generation is deployed, a distinction generic bandwidth planning tools frequently miss by treating “GPU count” as the only relevant scale variable.

Number of sites or campuses and growth horizon round out the baseline profile, with a value of 1 site used for a single-cluster deployment where the entire bandwidth question is scale-up only, and no cross-site scale-out demand exists at all.

Training versus inference is the primary driver of cross-site bandwidth need

The planner treats the training-to-inference workload split as the single most consequential input for cross-site DCI bandwidth specifically, because training synchronisation traffic is identified directly as the primary driver of cross-site bandwidth need — inference workloads, by contrast, generate comparatively light cross-site traffic limited mostly to replication and serving. A cluster running 100% inference has a fundamentally different DCI bandwidth profile than one running 100% training, even at identical accelerator count and generation, and the planner models that split explicitly via a slider rather than assuming a fixed ratio.

Model size band compounds this: small models under 10B parameters generate modest training bandwidth (roughly 10 Gbps) and minimal inference bandwidth, while frontier models over 1T parameters generate roughly 800 Gbps of training bandwidth demand and 50 Gbps of inference bandwidth — an 80x difference in training bandwidth need purely from model scale, independent of accelerator count or generation.

Distributed training pattern determines how much traffic actually crosses site boundaries

This is where the planner’s core technical insight lives: different parallelism strategies cross site boundaries very differently, and the difference is large. Data-parallel training, where full gradient all-reduce traffic crosses every site boundary, is identified as the heaviest cross-site synchronisation pattern, requiring round-trip latency under roughly 4ms for genuinely good performance and becoming marginal beyond roughly 10ms. Model-parallel training, where model shards mostly stay within a single site and only boundary activations cross, carries a 0.6x traffic factor relative to data-parallel and tolerates latency up to roughly 10-20ms. Pipeline-parallel training, where only activation and gradient hand-offs between pipeline stages cross site boundaries, carries just a 0.4x traffic factor and tolerates latency up to roughly 20-40ms. Inference-only workloads carry no training synchronisation traffic at all, with cross-site traffic limited to replication and serving, and correspondingly relaxed latency tolerance up to roughly 40-100ms.

This matters enormously for RFP specification: an organisation planning a data-parallel, multi-site training cluster needs dramatically tighter inter-site latency and higher bandwidth than one running the same accelerator count and generation but with a pipeline-parallel or inference-only workload — and specifying inter-site connectivity without knowing which parallelism strategy will actually be used risks either wildly over-provisioning or building a topology that can’t support the intended training approach at all.

Today’s topology is the baseline the gap analysis measures against

The planner asks directly about current scale-up fabric status (none, partial coverage across some racks or pods, or a full non-blocking fabric already in place), inter-site connectivity type (none, shared enterprise network, dedicated dark fibre, or leased wavelength service), current inter-site link capacity, and approximate round-trip latency between sites — with sensible defaults (0 capacity, 10ms latency) applied where the organisation doesn’t yet know these figures precisely. This baseline is what the entire gap analysis is measured against, since the planner’s output is fundamentally about the delta between what’s needed and what exists today, not an abstract bandwidth requirement in isolation.

Growth and redundancy calibration, applied to the right base

The final phase calibrates expected accelerator and cluster growth over the planning horizon and a redundancy or protection buffer for failover and burst capacity — both applied to today’s accelerator count and actual cross-site traffic volume, not to nominal link capacity, which is the technically correct basis for a genuine growth projection.

A deliberately bounded estimate, not an engineering spec

The planner is unusually direct that its bandwidth figures are illustrative, drawn from publicly published, generalised accelerator interconnect generations rather than tied to a specific hardware vendor’s product, and that latency feasibility thresholds are framed as a rule-of-thumb feasibility flag rather than a guaranteed engineering outcome. The disclaimer is explicit that this does not constitute network engineering, procurement, or investment advice, and that vendor-specific capacity planning and engineering validation — including from network and ML systems engineers — should be commissioned before any network design or procurement decision. That’s a genuine scope boundary: the tool exists to establish whether an RFP’s connectivity assumptions are in the right order of magnitude before expensive engineering work begins, not to replace that engineering.

From estimate to a defensible RFP starting point

The output — scale-up and scale-out bandwidth requirements, a latency feasibility check, gap analysis against current topology, and a growth headroom check — gives infrastructure teams a specific, sourced starting point for AI interconnect RFPs, rather than either a vendor’s generic bandwidth recommendation or an internal guess disconnected from the actual workload’s parallelism strategy.

Network planning teams, AI infrastructure architects, and system integrators scoping AI cluster connectivity can run the free calculator directly.

Related Tool: OTN Capacity & Right-Sizing Calculator
Once you’ve established your DCI bandwidth requirement, check how much of that inter-site capacity could come from right-sizing your existing optical transport network before committing to new equipment spend.