TL;DR
|
For a team renting GPU capacity in colocation or on bare metal, the InfiniBand vs RoCE for AI cluster decision can sit in a single line of the provider’s quote. That line decides which network carries training traffic between your servers. It also raises a question: which settings does the host keep right for as long as the cluster runs?
This guide is written for the buyer who rents the fabric, not the engineer who builds it. RoCE carries remote direct memory access (RDMA) traffic over Ethernet, while InfiniBand is a separate network technology.
The guide maps cluster sizes to what manufacturer reference designs show and explains where tuned RoCE has carried large training jobs. After that, it lists the evidence to ask a provider in India to produce. Neither fabric gets a recommendation here.
Our hub post on what makes a data centre GPU-ready for AI workloads covers power, cooling and networking together. This post goes one level down, into the network between GPU servers.
Which Network Does the Fabric Choice Affect?
The fabric choice affects the back end network, which carries training traffic between GPU servers. Meta describes a training cluster that relies on two independent networks, a front end and a back end. In its design, the front end network handles data ingestion, checkpointing and logging. The back end network handles training.
Meta built that back end as a non-blocking fabric. Its engineers say it gives high bandwidth, low latency and lossless transport between any two GPUs in the cluster. Storage, by contrast, sits on the front end network and feeds input data to the GPUs from there.
Inside a single server, GPUs use a different path again. NVIDIA’s DGX B200 reference architecture describes a system with eight B200 GPUs linked by NVLink and NVSwitch. The same design groups these systems into scalable units of 32.

Figure 1. Training traffic between GPU servers runs on the back end network, while storage, data ingestion and checkpoints use the front end network. The InfiniBand or RoCE choice applies to the back end network.
WHAT TO VERIFY
|
Once you know a job will cross servers, the next question is how each fabric is run.
How Do InfiniBand and RoCE Differ for a Renter?
Both fabrics carry RDMA traffic between GPU servers. They differ in the tools used to run the fabric. Each difference then becomes a question for the host.
NVIDIA’s DGX B300 reference architecture calls InfiniBand a low latency, RDMA capable networking technology. In that design, InfiniBand XDR has a peak speed of 800 Gbps per direction. So the manufacturer itself publishes both an InfiniBand XDR option and a Spectrum-X 2.0 option for the compute fabric of the same server.
RoCE takes a different approach. NVIDIA’s Cumulus Linux documentation explains that RoCE relies on PFC and ECN to operate. In lossless mode, the Ethernet layer handles congestion hop by hop. The same page notes that end hosts must support RoCE, and it shows how to check switch settings with the nv show qos roce command.
Meta’s SIGCOMM 2024 write-up describes retuning congestion control when it moved to 400G links. Default settings for Data Center Quantized Congestion Notification (DCQCN) degraded performance there, even with ECN thresholds doubled from its 200G networks. Its engineers then ran with PFC alone for over a year and observed stable performance for training collectives.
| Point to compare | InfiniBand | RoCE on Ethernet |
|---|---|---|
| What manages the fabric | NVIDIA designs manage the InfiniBand fabric with the Unified Fabric Manager appliance | PFC and ECN set on switches in lossless mode, with end hosts that support RoCE |
| Large-scale example | Meta cluster of 24,576 GPUs on Quantum2 InfiniBand | Meta cluster of 24,576 GPUs on RoCE |
| Setting to watch after an upgrade | Which fabric manager and firmware settings changed? | Congestion control and ECN thresholds, as in Meta’s 400G experience |
Sources: NVIDIA DGX B200 reference architecture; NVIDIA Cumulus Linux 5.16 documentation; Meta Engineering, March and August 2024.
The table compares two distinct operating models rather than presenting one as superior to the other. Meta has run large generative AI workloads on 24,576-GPU clusters in both columns. In the sources, RoCE lossless mode is configured through PFC and ECN settings on switches. NVIDIA’s InfiniBand designs, by contrast, include a Unified Fabric Manager appliance.
InfiniBand vs RoCE for AI Cluster Sizes: What the Reference Designs Show
Meta’s experience shows that cluster size alone does not settle the choice.
Meta built two clusters of 24,576 GPUs each, one on RoCE and one on NVIDIA Quantum2 InfiniBand. Both use 400 Gbps endpoints. Meta reports that careful co-design of network, software and model let it run large generative AI workloads on both without network bottlenecks. In March 2024, Meta described its Llama 3 training on the RoCE cluster as ongoing.
NVIDIA’s reference designs give a useful scale for smaller clusters. The DGX H100 SuperPOD network fabric guide counts the compute fabric switches for one to four scalable units. Its numbers appear in the table below, next to the question each size raises.
| Cluster size | Reference point | Question for the host |
|---|---|---|
| Eight-GPU node | NVLink and NVSwitch inside the chassis (DGX B200 design) | Will any job span two nodes? If not, does the quote include a back end fabric you will not use? |
| Part of one scalable unit | NVIDIA advises building leaf switches and cables for the full unit and leaving unused ports, for consistent performance | Is the fabric built for the full unit, or only for the nodes you rent? |
| One scalable unit | 31 nodes, 248 GPUs, 8 leaf and 4 spine switches (one node slot goes to the fabric manager) | What is the leaf to spine oversubscription ratio? |
| Two scalable units | 63 nodes, 504 GPUs, 16 leaf and 8 spine switches | Do your nodes share leaf switches with other tenants? |
| Four scalable units | 127 nodes, 1,016 GPUs, 32 leaf and 16 spine switches | What NVIDIA Collective Communications Library (NCCL) results has the host recorded at this size? |
| Tens of thousands of GPUs | 24,576 GPUs per cluster at Meta, on RoCE and on InfiniBand | Which team tunes congestion control, and what changed at the last speed upgrade? |
Sources: NVIDIA DGX H100 and DGX B200 SuperPOD reference architectures; Meta Engineering, March 2024.
Reading the sizing table
The table shows each step up adding leaf and spine switches. It does not show one fabric taking over at a set size. Within a scalable unit of 32 nodes, the reference design keeps traffic one hop within a rail. Traffic between rails crosses the spine layer, so how much spine capacity sits behind your leaf switches?

Figure 2. In NVIDIA’s DGX H100 design, a scalable unit of 32 nodes is rail aligned, so traffic stays one hop within a rail through the leaf switches. Traffic between rails crosses the spine layer. Where does each rented eight-GPU node sit in this layout?
Four buyer scenarios
- Fine-tuning on one eight-GPU node. GPU traffic stays on NVLink inside the server. Does the quote charge for a back end fabric this job would not use, and how fast do storage reads arrive over the front end network?
- Training across 4 to 16 nodes. This fits inside one scalable unit of the reference design. The host can be asked for NCCL test results at 4, 8 and 16 nodes on the exact fabric you would rent.
- Pre-training across one or more full units. NVIDIA publishes both an InfiniBand XDR and a Spectrum-X 2.0 compute fabric for DGX B300, built in units of 72 systems. Does the contract state the oversubscription ratio and the congestion settings?
- An inference fleet of single-node replicas. Will any model be split across servers? If none will, how much of the evaluation should go to front end bandwidth and storage instead?
Where the market is moving
Dell’Oro Group data, reported by SDxCentral, shows Ethernet at around two-thirds of data centre switch sales in AI clusters in Q1 2026. InfiniBand held 80 percent of the AI back end market in 2023. Even so, its switch sales more than tripled in Q1 2026. Then, in Q2 2026, AI back end switch sales passed front end sales for the first time, and Ethernet extended its lead.
Dell’Oro also expects the market to stay supply-constrained for at least the next one to two years. For a renter in India, that becomes a practical question. How long would a failed switch or optic take to replace at the site?
Ethernet also has a newer standards path. The Ultra Ethernet Consortium released its Specification 1.0 on 11 June 2025. It is an Ethernet-based stack that spans network interface cards (NICs), switches, optics and cables for multi-vendor use. So it is worth asking the host whether its back end runs RoCEv2, a UEC-based stack, or a mix.
With the fabric options and sizes mapped, the remaining step is to ask the host for proof.
What to Ask Your Host to Prove
Use verifiable evidence rather than relying solely on the fabric name mentioned in a quotation. The table below pairs each question with the reference point behind it and the document that answers it.
| Question | Reference point | Evidence to request |
|---|---|---|
| Which fabric and generation runs the back end network? | NVIDIA designs name the switch model and link speed per system | Switch and adapter models, link speed per GPU |
| What is the oversubscription ratio from leaf to spine? | NVIDIA’s DGX B200 SuperPOD compute fabric is non-blocking; Meta’s back end is non-blocking within an AI Zone | Topology diagram with port counts at each layer |
| Is the topology rail-optimised? | The DGX SuperPOD compute fabric is rail-optimised | Cabling map showing which leaf switch serves each rail |
| For RoCE, are PFC and ECN set on every hop? | Cumulus Linux lossless mode uses both | Switch output of nv show qos roce, and confirmation that end hosts support RoCE |
| For InfiniBand, who runs the fabric manager? | NVIDIA designs use the Unified Fabric Manager appliance | A named owner, plus read access to fabric health reports |
| What do NCCL tests show at your node count? | nccl-tests reports a busbw column for each message size | all_reduce_perf output at 1, 2 and your target node count |
| What changed at the last speed upgrade? | Meta saw degraded performance with default settings at 400G | Change log of congestion settings and the retest results |
| How long does a spare switch or optic take to arrive? | Dell’Oro expects supply constraints for one to two years | Spares held on site and replacement lead times |
Sources: NVIDIA DGX SuperPOD reference architectures; NVIDIA Cumulus Linux 5.16 documentation; NVIDIA nccl-tests; Meta Engineering, August 2024; Dell’Oro Group via Lightwave, September 2026.
Testing the fabric with NCCL
The NCCL row asks for measured results. The nccl-tests README shows a 64-GPU run across 8 nodes with mpirun -np 64 -N 8 ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1. It also points to a performance page that explains the busbw column.
The host can be asked to run the same test on the nodes you would rent. The one-node result then gives a baseline for the result at your full node count.
| EVIDENCE TO REQUEST A topology diagram with the leaf to spine oversubscription ratio. Lossless settings output for RoCE, or fabric manager reports for InfiniBand. nccl-tests all_reduce_perf results at your exact node count. The change log from the last link speed upgrade. |
Where the Fabric Choice Does Not Help
A fabric decision leaves other limits in place. In Meta’s design, the front end network carries data ingestion and checkpoints. Does storage on that side have its own tests? Rack power and heat are separate questions too.
Meta’s own caveat also applies to any copied design. Its RoCE tuning depends on coordination between its collective library and network, and Meta says it may not apply to all scenarios. Do the host’s test results come from a cluster of the same size and fabric as the one you would rent?
| Criterion | Why it matters | What to verify |
|---|---|---|
| Job span | The back end fabric carries training traffic between servers | Will any job cross nodes? |
| Oversubscription | NVIDIA’s DGX SuperPOD compute fabric is described as non-blocking, and so is Meta’s back end within an AI Zone | The ratio, in writing |
| Fabric operation | RoCE lossless mode uses PFC and ECN; NVIDIA designs manage InfiniBand with the Unified Fabric Manager appliance | Configuration output or fabric reports |
| Measured collectives | nccl-tests reports bus bandwidth as busbw | Results at your node count |
| Spares | Dell’Oro expects a supply-constrained market | Lead times for switches and optics |
FAQs
Is RoCE good enough for large language model training?
Meta’s March 2024 post describes Llama 3 training as ongoing on its RoCE cluster of 24,576 GPUs. Its August 2024 post says its RoCE networks support large models such as Llama 3.1 405B. Meta credits careful co-design of network, software and model for running large workloads without network bottlenecks. Can the host show test results at your node count?
What oversubscription ratio should a GPU training cluster have?
NVIDIA’s DGX B200 SuperPOD architecture uses a non-blocking compute fabric, while Meta’s AI Zone architecture uses a non-blocking back-end network within each AI Zone, with oversubscription intentionally introduced between AI Zones. The specified back-end oversubscription ratio and its comparison with NVIDIA’s reference architecture should be clearly documented in the network design and contract.
Does InfiniBand need its own management tools?
In NVIDIA’s reference designs, the Unified Fabric Manager appliance manages the InfiniBand fabric. The DGX H100 design even gives up one node slot per scalable unit to make room for it. When renting, the questions are who operates it and which reports you can read.
Will Ultra Ethernet replace RoCE in GPU clusters?
The Ultra Ethernet Consortium released Specification 1.0 in June 2025, introducing an Ethernet-based networking stack designed for AI and high-performance computing (HPC) workloads. Its adoption in GPU clusters depends on hardware compatibility and feature availability. Current switch and adapter support, along with features that remain on the development roadmap, should be clearly outlined to assess its readiness as an alternative to RoCE.
Choosing a Fabric You Can Verify
Meta has run large generative AI workloads on 24,576-GPU clusters of both kinds. For a buyer, the sharper question is which fabric the host can show running well at your size. If every job fits in one node, is a back end fabric needed at all? When jobs cross servers, the evidence in this guide is worth having before quotes are compared.
A host that answers with topology diagrams, configuration output and NCCL results gives you something to check. One that answers only with a fabric name gives you the same question again.
Related reads go beyond the fabric. Our guide to air vs liquid cooling for GPU racks sizes rack power and heat by kW per rack. If you are weighing virtual GPU instances against dedicated servers, our post on bare-metal performance for GPU workloads covers that layer. Our earlier post covers the role of InfiniBand in RackBank’s AI infrastructure.
| NEXT STEP Planning GPU racks and want to put these fabric questions to a provider? See RackBank AI Colocation for high-density GPU racks, power and cooling. |