Deep Dive Into a Network designed for AI workloads with CoreWeave

Min Jun, the Director of Product for networking, oversees all aspects of CoreWeave’s networking infrastructure, including data center front-end and back-end networks, isolation, security, and the backbone and edge. The presentation focuses on the critical networking components that enable high-performance AI training and inference. CoreWeave leverages a comprehensive approach, encompassing both back-end and front-end networks, along with specialized hardware like DPUs, to deliver scalable and efficient solutions for AI workloads.

The core of CoreWeave’s AI infrastructure lies in its back-end network, essential for high-bandwidth GPU-to-GPU communication during AI training. CoreWeave employs both RoCE (RDMA over Converged Ethernet) and InfiniBand technologies. Their RoCE network is a two-tier, multi-plane, multi-rail system designed to scale to over 120,000 GPUs in a single fabric, optimized for efficient collective operations. A notable innovation is the integration of liquid cooling for all RoCE back-end switches, sharing the same software-defined infrastructure as the compute, which significantly reduces rack footprint and power draw, particularly beneficial for high-scale GPU deployments. Alongside RoCE, CoreWeave also deploys large InfiniBand networks, providing a battle-tested option with even lower latency and features like SHARP for in-network collective operations. While InfiniBand is excellent for smaller scales, CoreWeave offers dedicated expert support to ensure seamless deployment and utilization. DPUs, specifically BlueField, run the front-end network on every node, creating a clear delineation between the customer-owned host and CoreWeave’s infrastructure. These DPUs offload networking functionalities such as firewalling, IPAM, and VPCs, freeing up customer compute resources and laying the groundwork for future innovations like inference and agentive workflows. GPUs thus have three network paths: NVL72 within a rack, ConnectX NICs for RoCE/InfiniBand for cross-rack communication, and DPUs for north-south traffic to the internet or storage.

Further enhancing its infrastructure, CoreWeave recently unveiled the first global backbone network purpose-built for AI. This extensive network circumnavigates the Earth, with reach extending to Asia Pacific, the Middle East, North America, and Europe. Operating at a petabit scale, this global backbone allows CoreWeave to strategically provide AI capacity precisely where customers need it, supporting diverse and demanding AI workloads across different geographies.