Network Engineer Meet AI Cluster with Upscale AI

The session provides a worldview of AI clusters and the importance of the network infrastructure behind them. Aninda Chatterjee of Upscale AI tackles the challenge network engineers face in understanding complex AI clusters. It begins by contrasting traditional data center networking, characterized by north-south traffic patterns and three-tier architectures with spanning tree, against the evolution driven by virtualization, microservices, and distributed storage, which shifted traffic predominantly east-west, leading to the adoption of the Clos architecture. The core shift from CPU-centric computing, often bottlenecked by memory channels, to GPU-centric architectures with thousands of tensor cores and high-bandwidth memory (HBM) is highlighted. This shift creates new demands: GPU memory is fast but small, so models must be split across GPUs, requiring substantial inter-GPU communication to exchange partial results and reductions, which places considerable strain on the network infrastructure.

The discussion then delves into the three tiers of AI backend cluster networking: “scale-up” (intra-rack communication with sub-microsecond latency, acting as one large GPU), “scale-out” (inter-rack communication using Leaf-Spine architectures, Rocky V2, Ultra Ethernet, or InfiniBand, with microsecond latency), and “scale-across” (data center interconnects for stitching sites into a single logical AI factory). A key architectural concept introduced is “rails,” where specific NICs from different GPUs connect to the same leaf switch to optimize communication for certain collective operations, minimizing hops for intra-rail traffic while using spines for cross-rail communication. Beyond the backend, the crucial roles of the front-end and storage networks are emphasized, handling long-lived streaming connections, pulling large model weights (often hundreds of gigabytes), storing training checkpoints, and managing KV cache offloading, which involves different traffic patterns and resiliency requirements compared to the backend.

The inference process, broken into pre-fill (compute-bound, generating the first token) and decode (memory bandwidth-bound, generating subsequent tokens iteratively), further illustrates network demands. The KV cache, storing keys and values of earlier tokens to avoid recomputation, can grow significantly, necessitating offloading to storage or other memory pools, thereby increasing network traffic between GPUs and storage. The presentation concludes by underscoring that effectively building and scaling AI infrastructure requires a deep understanding of what data is moving, how often, and who is waiting, whether through tensor, pipeline, or expert parallelism, or even disaggregated serving where GPUs specialize in pre-fill or decode, creating handoff traffic for the KV cache across the network.