Introduction

Legacy data center networking are optimized for random “mice flows”, allowing hardware to exploit statistical multiplexing gains. In this model, the probability of every port bursting toward a single destination at the exact same microsecond is low enough to be ignored. AI training operations challenges this assumption.

When a 1,024-GPU cluster completes a backward pass, it transitions instantly from idle to a high-bandwidth, low-entropy step function. Because the traffic is deterministic and synchronized, the fabric is hit with massive “elephant flows” that target the same aggregation points simultaneously. This creates a deterministic incast that collapses switch buffers in windows of microseconds. Standard monitoring tools, which aggregate data over seconds, remain blind to these micro-bursts, reporting healthy utilization while the fabric is actually paralyzed by Go-Back-N retransmission storms and PFC pause trees.

As AllReduce grows to occupy up to 65% of total training time, the network is now a primary determinant of JCT. And as training workloads grow this makes high-performance networking crucial.

Source: Usenix

AllReduce Breaks Every Assumption in buffer sizing

From the vantage point of a ToR switch, Ring-AllReduce manifests as a temporal singularity. Upon the completion of the backward pass, the fabric undergoes a state transition from idle to line-rate saturation via a Heaviside step function. This synchronized injection occurs in less than a microsecond, collapsing switch buffers long before DCQCN hardware feedback loops can modulate the injection rate.

At the moment the backward of Ring-AllReduce pass completes, every NIC on every server connected to the ToR switch begins transmitting simultaneously at line rate, and every packet is destined for the same upstream aggregation paths. The switch, designed around the assumption that traffic arrives as a Poisson process, suddenly faces a traffic pattern that violates its buffer sizing model. This is incast.

We can illustrate this with an hypothetical NVIDIA training model formed by an NVIDIA DGX H100 connected to 8 NICs a up to 400G representing a total 3.2 Tbps of aggregate bandwidth at the access layer.

Source: NVIDIA

If we assume one Scale Unit of 32 servers or nodes operation, that give us 102.4 Tbps, considering the switch for a server with a requirement of 400Gb/s InfiniBand/Ethernet , the Quantum2 9700 complies but only has 51.2 Tbps, which implies a 2:1 oversubscription ratio.

Now, in a 8 switch arrangement the system becomes non-blocking, the potential total bandwidth per switch now is 102.4Tbps/8 = 12.8 Tbps.

Source: NVIDIA

By using an 8-switch arrangement, the system becomes non-blocking, meaning there is enough internal capacity for all traffic to flow without stopping. This gives each switch a massive bandwidth of 12.8 Tbps.

However, even with that speed, a problem called Incast can occur. If 32 servers try to send data to the same destination port at the exact same time. Because the QM9700 uses a shared buffer, this sudden “traffic jam” can overwhelm the port’s memory. To fix this, the system uses SHARPv3 technology. Instead of letting the packets crash at the exit, the switch becomes the destination and processes the data itself. For larger messages (over 16–64 KB), it triggers the SAT (Streaming Aggregation Technology) protocol. The trade-off is that the switch must break these large messages into multiple SHARP segments. While this prevents the network from clogging, it adds coordination overhead and increased latency.

Source: NVIDIA

Enterprise ToR switches allocate shared packet buffers using a model derived from general-purpose east-west server traffic. Traffic arrives in a distribution close enough to Poisson that a modest shared buffer can serve as an overflow reservoir for bursty flows with high statistical efficiency. A single port might burst, but the probability that all 48 ports burst simultaneously is low enough that the shared pool is almost never fully consumed.

Under AllReduce incast, that assumption does not degrade gracefully. It collapses immediately, on the first collective operation of the first training job.

At 400 Gbps per port, each transfer at a speed of 50 GB/s. Across the entire 8-switch fabric, the aggregate ingress is 12,800 GB/s.

32 nodes * 8 * (50 GB/s) = 12,800 GB/s

Or 102.4 Tb/s , as our example assumes 8 switches at a throughput capacity of 51.2 Tb/s , or 25% capacity.

Under normal operation the 8-switch fabric distributes load evenly. The failure mode is egress contention when a synchronized AllReduce burst injects into the switch. If we consider that AI training traffic patterns are characterized by massive, sustained “elephant flows” of data, using east-west traffic,

32 * (50 GB/s) = 1,600 GB/s or 1.6 MB/µs

As the Quantum 2 has 64 MB of shared buffer , the temporal window is:

64MB / (1.6 MB/microsecond)= 40 µs

This is faster than reaction time DCQCN (Data Center Quantized Congestion Notification) algorithms, which operate at +100 microsecond range for hardware-level feedback loops, which creates a “blind spot” in network defense in DCQCN and PFC mechanisms.

Source: Aruba

Past the point of the 40 microsecond buffer saturation, the legacy switch enters a reactive state governed by one of three mechanisms. In a general-purpose cloud environment, these protocols provide a safety net for lossy traffic; however, in the context of barrier-synchronized AI collectives, each mechanism is fundamentally incompatible with the requirement for deterministic, low-jitter transport. Whether the switch chooses to drop, pause, or probabilistically discard, the result is a catastrophic divergence between physical wire throughput and application-level Goodput.

The AllReduce Tax

For RoCE v2 tail drop is catastrophic. RDMA is designed for lossless networks. Its reliability mechanism uses GBN retransmission reliability model. When a dropped packet is detected, the sender does not retransmit only that packet. It retransmits every packet sent after the dropped one.

In a 400G flow that has been running for 100 microseconds before the drop occurs — a realistic window given the 1.28 ms buffer fill time, the sender has transmitted approximately:

400 Gbps × 100 µs = 5 MB of gradient data

Every byte of that 5 MB must be retransmitted in its entirety under Go-Back-N, even though only one or a few packets in that window were actually dropped. Across 128 simultaneous flows all experiencing tail drop at the same buffer overflow event, the aggregate retransmission volume is:

128 flows × 5 MB = 640 MB

This creates a multiplicative load amplification effectively turning the fabric into a “retransmission engine” where the wire is saturated but Goodput hits zero. Because of GBN’s inefficiency, a deterministic packet loss rate as low as 0.4% can collapse application-level goodput to zero, as the sender retransmits entire sequences of packets for every single drop.

The per-iteration figure compounds across the backward pass in a way the flat retransmission volume obscures. A frontier-scale transformer at GPT-3 175B scale uses 96 layers.

Source: Open AI

A Megatron-LM’s tensor parallelism scheme requires two AllReduce operations per transformer layer in the backward pass (one synchronizing the MLP block gradients and one synchronizing the attention block gradients). That is 192 AllReduce synchronization events per optimizer step, each one a independent incast trigger capable of collapsing the switch buffer if the fabric has not recovered from the previous event.

Source: NVIDIA

If we assume the case of GPT-3 175B, we have:

Total tokens: 300,000,000,000 / (Batch size: 3,200,000 tokens/step) = 93,750 Steps

Taking 2 AllReduce operations per layer per backward pass for Megatron-LM tensor, current market data of H100 spot rate, a retransmission duration per AllReduce event of around 100 µs and taking a typical cluster size of 1,024 GPUs. We have our stall costs per AllReduce event.

Source: Bloomberg

For the utilization rate, we consider OpenAI’s own training run for GPT-3 achieved 19.6% MFU. Two years later, Google achieved 46.2% MFU training PaLM at 540 billion parameters on TPUv4, with network communication overhead cited as the primary efficiency loss at scale. On H100 hardware with mature software stacks, Llama 3.1 reached around 38 to 43% MFU. If we assume a MFU range of 20-45% that leaves us with a stranded cost around $1.0M–$1.8M retransmission losses as we estimate in the following graph.

The Pause That Stops a Thousand GPUs

PFC attempts to maintain a lossless fabric by issuing PAUSE frames (IEEE 802.1Qbb). However, in a 1,024-GPU AllReduce collective, the hard synchronization property means that a single PAUSE frame sent to one NIC on one ToR switch stalls the entire global job. This induces HOL blocking and “Congestion Trees,” where a hotspot in one rack silences innocent senders across the entire Pod, stranding millions in compute capital to solve a micro-burst on a single port.

A standard PFC pause duration is 65,535 quanta at 100G, equal to approximately 0.67 ms. In a barrier-synchronized 1,024-GPU AllReduce collective, that 0.67 ms pause on a single port idles all 1,024 GPUs simultaneously, the synchronization guarantee that makes AllReduce correct is the same property that makes a single pause event cluster-wide. At $2.50–$3.00/GPU-hour, 0.67 ms of forced cluster idle costs $0.0476¢ – $0.0572¢ per PAUSE event. Under sustained incast, PFC pause events cascade as backpressure propagates upstream through the congestion tree, multiplying that per-event cost across every switch tier in the pod.

WRED: The Congestion Avoidance Trap

Because RoCEv2 relies on the Go Back N (GBN) reliability model, a single “early” drop by WRED triggers the same catastrophic retransmission storm as a full tail drop. In a 400G environment, this forced pullback of megabytes of already acknowledged data creates a multiplicative load, driving a sudden retransmission storm that consumes the very buffer space WRED was designed to protect.

These mechanisms, specifically Tail Drop, PFC PAUSE, and WRED, often activate simultaneously during the 40 microsecond buffer collapse of a synchronized AllReduce burst. The result is a progressive failure where localized congestion propagates through PFC pause trees and GBN feedback loops, stalling 1024 GPU clusters to resolve a single port incast event. This microburst volatility is why GPT4 scale clusters often see GPU utilization languish between 32% and 36%. The network is not merely a transport; it operates as a barrier synchronized bottleneck where the slowest packet defines the ROI of the entire capital investment.

In a 4-uplink topology, a single hash collision that pins one link at 100% utilization while the remaining three sit at 0% results in a dashboard reporting 25% aggregate fabric utilization. Even under moderate imbalance, standard monitoring that averages across uplinks can report ~43% total utilization, failing to trigger threshold-based alerts while specific ports are in persistent PFC pause states.

Thanks for reading Bantiv! This post is public so feel free to share it.

Share

References

1. Y. Zhong et al., “DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving,” arXiv preprint arXiv:2401.09670, 2024.
2. Y. Zhu et al., “Congestion Control for Large-Scale RDMA Deployments,” in Proc. ACM SIGCOMM, London, UK, 2015, pp. 523–536.
3. Y. Li et al., “HPCC: High Precision Congestion Control,” in Proc. ACM SIGCOMM ‘19, Beijing, China, 2019, pp. 44–58.
4. Broadcom, “A Look At Broadcom’s Jericho3-AI Ethernet Fabric: Schedules, Credits, And Cells,” Network Field Day 32, Aug. 2023.
5. OrhanErgun.net Blog, “AI-Powered Network Optimization: Case Studies and Results,” by J. Lake, Jan. 2026.
6. CoreWeave, “Achieve AI Infrastructure Goodput of up to 96% with 3 Key Strategies,” CoreWeave Blog, 2026.
7. X. Han et al., “Avoid Routing Polarization for OCS-based GPU Clusters,” arXiv preprint arXiv:2603.28168, 2026.
8. NVIDIA, “NVIDIA Spectrum-X: The Ethernet Designed for AI,” White Paper, 2024.
9. GFI Software, “Monitoring interface throughput,” Exinda Network Orchestrator Help System, 2021.
10. Y. Lei et al., “Nanosecond Precision Time Synchronization for Optical Data Center Networks,” arXiv preprint arXiv:2410.17012, 2024.
11. M. Pitr, “Disaggregated Serving: Achieving consistent online LLM inference performance,” Paper Journal: From Scratch Substack, Mar. 16, 2025.
12. Broadcom, “StrataDNX Jericho3AI Machine Learning Ethernet Switch BCM88890 Data Sheet,” 2023.
13. Brian, “Understanding NVIDIA’s Spectrum-X Solution,” Fibermall.com Blog, Oct. 17, 2024.
14. Z. Wang et al., “Revisiting SLO and Goodput Metrics in LLM Serving,” arXiv preprint arXiv:2410.14257, 2024.
15. H. Atmer et al., “Prefill vs. Decode Bottlenecks: SRAM–Frequency Tradeoffs and the Memory-Bandwidth Ceiling,” arXiv preprint arXiv:2512.22066, 2025.
16. A. Varre et al., “Low-Entropy Outputs of Softmax,” arXiv preprint arXiv:2603.06248, 2026.
17. J. Hu et al., “Load Balancing in PFC-Enabled Datacenter Networks,” in Proc. 6th Asia-Pacific Workshop on Networking (APNet ‘22), Fuzhou, China, 2022, pp. 1–8.
18. NVIDIA, “NVIDIA Spectrum-X Network Platform Architecture: The First Ethernet Network Designed to Accelerate AI Workloads,” White Paper, 2024.
19. P. Jacquet, T. Ledoux, and R. Rouvoy, “ScroogeVM: Boosting Cloud Resource Utilization with Dynamic Oversubscription,” IEEE Transactions on Sustainable Computing, 2024.
20. NVIDIA, “NVIDIA DGX H100 Data Sheet,” 2022.
21. B. Spang, S. Arslan, and N. McKeown, “Updating the Theory of Buffer Sizing,” in Proc. IFIP Networking Conference, 2021.
22. C. Guo et al., “RDMA over Commodity Ethernet at Scale,” in Proc. ACM SIGCOMM, Florianopolis, Brazil, 2016, pp. 202–215.
23. D. Gibson et al., “Aquila: A unified, low-latency fabric for datacenter networks,” in Proc. USENIX NSDI, 2022.