HPE Networking Data Center Exploring Evolving Congestion Management in Scale-Out Networks

Managing congestion in AI training and inference fabrics is critical to maximize GPU utilization, minimize latency and jitter, and protect model accuracy and throughput. Ultimately, congestion-free networks are required to deliver the best end user experiences in the era of agentic AI. This session examines the evolution from dynamic and global load balancing to RDMA-aware load balancing, intelligent packet spraying, and emerging Ultra Ethernet mechanisms. At Networking Field Day 41, Praful Lalchandani, Vice President of Product Management for Data Center Networking at HPE Networking, led the session.

Lalchandani explained that AI fabrics carry a small number of very large flows, so static ECMP hashing leaves some links idle while others drop packets. He split the solution into congestion avoidance through better load balancing and congestion control through fast signaling. HPE’s pre-UEC avoidance work began with dynamic load balancing, which chooses paths based on link utilization and queue depth in hardware. Global load balancing extended this to end-to-end path quality, much like a navigation app. HPE then introduced RDMA-aware load balancing, which splits an RDMA transfer into subflows identified by queue pair and pins each one to a specific spine using BGP link coloring, an approach HPE published through the IETF. In testing, RLB came close to the theoretical maximum of about 370 GB/s with consistent results across runs and eliminated ECN, PFC, and CNP events. If a link fails, only the flows on that link fall back to dynamic load balancing.

Lalchandani said HPE has tested Ultra Ethernet Transport on its QFX switches but has not yet seen production-grade UET implementations on NICs. HPE has, however, adopted UEC concepts such as packet trimming and selective acknowledgment for RDMA traffic. He sees the market moving toward MRC, which borrows UEC’s congestion techniques but keeps the standard RDMA frame format. MRC takes advantage of multiplane fabrics through ECMP-based spraying or SRv6 path pinning, an approach he likened to RLB. Tests with AMD MI350 GPUs, Pollara NICs, and 800G QFX switches put MRC ahead of DLB and ECMP, while SRv6 testing is still underway. He challenged vendor claims of 1.6x or 1.8x Ethernet performance gains, arguing that those figures compare against static ECMP. In his view, open-standard Ethernet already approaches theoretical maximums, and HPE publishes its benchmark results in public validated designs.