Watch on YouTube
Watch on Vimeo
CoreWeave is the essential cloud for AI, distinguishing itself by building infrastructure specifically optimized for the unique and demanding requirements of AI workloads, unlike traditional data center deployments or general-purpose clouds. Sean Derrington introduced CoreWeave’s portfolio, operations, compute, storage, and networking infrastructure, emphasizing how they approach these elements differently to address AI’s specific needs. He also highlighted CoreWeave Forge, an end-to-end platform designed as a continuous loop for running, evaluating, measuring, improving, and curating AI models, whether open or closed, with this particular presentation focusing on the underlying infrastructure. Mission Control, another CoreWeave offering, provides operational insights and an agent for natural language interaction with models to optimize GPU storage and performance.
On the compute front, CoreWeave showcased its liquid-cooled data centers featuring GB200 and Vera Rubin racks, and holds patents for its liquid-cooling control system designed to prevent large spills. They were the first to deliver NVIDIA Vera Rubin systems, validate the architecture with Dell, release the DeepSeek R1 benchmark, and have a customer, Cognition, use Vera Rubin in production, reporting 3-5x performance improvements. For storage, CoreWeave AI Object Storage (CAIOS) is S3-compliant, offers unlimited capacity, and is optimized for AI workloads with multi-cloud capabilities. A key innovation is LODA (Local Object Transport Accelerator), a node-level NVMe caching mechanism providing 7 GB/s throughput and dramatically reducing read latency by up to 8x, with a leading model provider achieving 99.7% cache hit rates across 15,000 GPUs. All CoreWeave object storage tiers (hot, warm, cold) are SSD-based with intelligent auto-tiering by age of access, plus an optional, lower-cost hard drive-based archiving tier. CoreWeave’s billing model for object storage is also unique, featuring no egress, operation, or early deletion fees, and offering programs to offset migration costs from other clouds.
CoreWeave’s networking infrastructure is purpose-built for AI, utilizing InfiniBand and RDMA for front-end connectivity to minimize latency and ensure rapid data transfer to GPUs. The back-end network is non-blocking and non-oversubscribed, facilitating high-throughput communication between GPUs, particularly with NVIDIA’s NVLink capabilities. Dedicated NICs for application and storage traffic prevent resource contention, ensuring optimal performance for AI-specific machine types. Security is enhanced with DPUs (Bluefield 3 or 4) on every node, offloading networking and security operations to free up CPU cycles. Globally, CoreWeave operates over 51 data centers with 1.5 gigawatts of power in North America and Europe, and has expanded its global AI network to Asia Pacific and the Middle East with announced plans in India and Indonesia, and network locations in Japan, Australia, and Dubai. This global backbone features dual 400 gig links and MACSec encryption, designed to handle “elephant type flows” by providing substantial bandwidth for rapid data movement between regions. Users can experience this infrastructure through “Arena,” a 30-day trial featuring the latest GPUs, and CoreWeaveForge also offers a free sign-up for model testing.
Personnel: Sean Derrington
Watch on YouTube
Watch on Vimeo
Harshdeep Singh Banwait, CoreWeave’s Senior Director of Compute, works with customers to understand their evolving workloads and collaborates with data center teams on infrastructure operations for all CPU and GPU instances. He highlighted recent innovations, noting that CoreWeave was one of the first cloud providers to bring up and rigorously validate an entire Vera Rubin rack. This involved comprehensive checks of individual servers, backend networking, power shelves, and NVLink switches to ensure proper system function. CoreWeave released results on DeepSeek R1, showing a 10x improvement in tokens-per-second throughput between GB200 and Vera Rubin instances, then partnered with Cognition as the first customer on Vera Rubin, achieving 4-5x performance improvements for their custom reinforcement learning and inference models.
The Vera Rubin rack ecosystem is complex, integrating six different chips: the new Ruby GPU, NVIDIA’s second-generation ARM CPUs (VR CP), and new NVLink six switches, which enable GPUs within a rack to communicate at 3 terabytes per second, effectively making the entire rack operate as a single brain. CoreWeave also upgraded its backend networking with ConnectX9 SuperNICS and new Spectrum X. A key innovation is the liquid-cooled Bluefield 4 DPU on every compute instance, managing the Kubernetes service to give customers full, bare-metal access and avoid virtualization overhead. Furthermore, CoreWeave custom-designed “Valvey” (unofficially Valvey McValveTrain), a patent-pending valve assembly for liquid-cooled racks that enables per-rack cooling control and a rapid 7-second emergency shutdown. Complementing this is “Racky,” an in-rack manager that collects temperature, leak, and flow data, allowing independent rack management even if compute is down. The company also developed a rack lifecycle controller to streamline provisioning and ongoing management of rack-scale compute, automating tests for components ranging from cooling units and power to firmware and multi-node functionality, ensuring high performance and reliability.
CoreWeave provides customers with a comprehensive dashboard, dubbed “Mission Control,” offering cluster-wide and rack-level visibility into performance, GPU utilization, temperature, and NVLink utilization. This allows customers to identify if performance bottlenecks stem from their application code or potential hardware issues, which CoreWeave’s teams actively manage and resolve. The platform’s bare-metal isolation and 100 Gbps front-end networking are crucial for agentic AI workloads, where CPUs and GPUs engage in continuous back-and-forth communication: GPUs for “thinking” and CPUs for “executing” actions like API calls or opening browser windows. CoreWeave supports CPU-only instances with Intel Emerald Rapids, AMD Turin and Genoa, and upcoming NVIDIA Vera, all built to rigorous performance and reliability standards. While generally providing optimized “best recipes” for firmware and drivers, CoreWeave can accommodate specific versions for sufficiently large customer clusters.
Personnel: Harshdeep Singh Banwait
Watch on YouTube
Watch on Vimeo
Deok Filho, Product Manager for Sandbox at CoreWeave, discusses the emergence of sandboxes as a crucial new compute primitive for running AI agents. He highlights that while containers have been used for a long time with controls over networking, storage, and identity, agents, which have become proficient in complex tasks like booking tickets or writing and deploying code, require a new level of containment. This necessity arises from agents performing “unintentional things,” leading to security concerns and the need for a compute primitive that offers more granular control than traditional serverless container solutions like AWS Lambda or Google Cloud Run, which are not designed for the dynamic, often untrusted code generated by agents.
The core difference in a sandbox, as offered by CoreWeave, is its ability to provide extremely granular control over an agent’s environment. Unlike serverless functions that run repetitive, known code, agents generate novel, untrusted code that requires robust isolation and specific governance. CoreWeave employs a Kata-based microVM container runtime, not OpenShell, to achieve this, building granular controls for networking, storage, and identity on top. This allows for precise restrictions, such as limiting an agent’s access to specific API endpoints or CIDR ranges. Sandboxes primarily utilize CPUs for both training (especially reinforcement learning environments) and production inference, addressing an unforeseen surge in demand for CPU capacity. Key use cases include coding agents, autonomous agents for QA or cybersecurity testing, RL environments for model post-training, and CI/CD, where agents produce high volumes of code.
CoreWeave’s solution addresses customer needs for useful capacity, reliability, and reduced operational overhead. They leverage existing CPU cores on GPU nodes (which often sit idle during GPU-heavy training) to run sandboxes, maximizing utilization of reserved compute. For bursty agent workloads, CoreWeave has developed a multi-zonal capacity management system. This system intelligently fills a customer’s reserved nodes first, then utilizes spot nodes, and finally spills over to CoreWeave’s serverless offering for peak demand, ensuring continuous service while managing costs. Features include fast snapshot and restore for agent trajectories, persistent distributed file systems for agent context and memory, and network rules with SNI filtering to enforce guardrails across a cluster. CoreWeave acknowledges that fine-grained security features like per-action policy enforcement or secrets injection often require external AI gateways, although they are building some of these capabilities into their own platform and plan to offer GPU sandboxes soon.
Personnel: Deok Filho
Watch on YouTube
Watch on Vimeo
Chris Milsted, a specialist field engineer, explained that CoreWeave’s primary goal is to keep expensive GPUs as active and close to 100% utilized as possible, especially as modern GPUs like the H100 and Vera Ruben deliver rapidly increasing bandwidth. These systems demand immense data speeds, with front-end network connections around 200 gigabits, posing a significant challenge for efficient data delivery. The presentation aims to demonstrate how CAIOS, LOTA (Local Object Transport Accelerator), and DFS (Distributed File System) provide a significant advantage in meeting these demands.
LOTA leverages local NVMe disks within each GPU node (e.g., eight 7.62 TB NVMe drives, with six dedicated for cache, providing 40-41 TB per node) to transparently cache data. When a user or workload accesses an object for the first time, it is pulled from the backend CoreWeave AI Object Storage into the local cache. Subsequent accesses are then served directly from this high-speed local cache, drastically reducing latency. While currently a read-only cache, CoreWeave also offers cross-region write acceleration, allowing local writes to be buffered to a local object storage array and then asynchronously destaged to a remote, durable object storage, addressing challenges for multi-region deployments or large checkpoint files.
This technology solves critical problems such as accelerating access to massive datasets, which can span petabytes or exabytes, and enabling fast data access across different data centers without needing to create costly duplicate copies of entire datasets. LOTA operates transparently, meaning users only need to use a different endpoint without needing to manage the cache directly. A demonstration showcased significant improvements in time to first byte when data was served from the local cache compared to pulling it across transatlantic links from remote object storage. Additionally, CoreWeave’s MCP offers easy monitoring and debugging of cache performance, while DFS provides a low-latency alternative for file system protocols.
Personnel: Chris Milsted
Watch on YouTube
Watch on Vimeo
Min Jun, the Director of Product for networking, oversees all aspects of CoreWeave’s networking infrastructure, including data center front-end and back-end networks, isolation, security, and the backbone and edge. The presentation focuses on the critical networking components that enable high-performance AI training and inference. CoreWeave leverages a comprehensive approach, encompassing both back-end and front-end networks, along with specialized hardware like DPUs, to deliver scalable and efficient solutions for AI workloads.
The core of CoreWeave’s AI infrastructure lies in its back-end network, essential for high-bandwidth GPU-to-GPU communication during AI training. CoreWeave employs both RoCE (RDMA over Converged Ethernet) and InfiniBand technologies. Their RoCE network is a two-tier, multi-plane, multi-rail system designed to scale to over 120,000 GPUs in a single fabric, optimized for efficient collective operations. A notable innovation is the integration of liquid cooling for all RoCE back-end switches, sharing the same software-defined infrastructure as the compute, which significantly reduces rack footprint and power draw, particularly beneficial for high-scale GPU deployments. Alongside RoCE, CoreWeave also deploys large InfiniBand networks, providing a battle-tested option with even lower latency and features like SHARP for in-network collective operations. While InfiniBand is excellent for smaller scales, CoreWeave offers dedicated expert support to ensure seamless deployment and utilization. DPUs, specifically BlueField, run the front-end network on every node, creating a clear delineation between the customer-owned host and CoreWeave’s infrastructure. These DPUs offload networking functionalities such as firewalling, IPAM, and VPCs, freeing up customer compute resources and laying the groundwork for future innovations like inference and agentive workflows. GPUs thus have three network paths: NVL72 within a rack, ConnectX NICs for RoCE/InfiniBand for cross-rack communication, and DPUs for north-south traffic to the internet or storage.
Further enhancing its infrastructure, CoreWeave recently unveiled the first global backbone network purpose-built for AI. This extensive network circumnavigates the Earth, with reach extending to Asia Pacific, the Middle East, North America, and Europe. Operating at a petabit scale, this global backbone allows CoreWeave to strategically provide AI capacity precisely where customers need it, supporting diverse and demanding AI workloads across different geographies.
Personnel: Min Jun
Thank you for being part of the Tech Field Day community! Our mailing list is a great way to stay up to date on our events and technical content, and we appreciate your signup.
We promise that we’ll never spam you, send ads, or sell your information. This list will only be used to communicate with our community about our events and content. And we’ll limit it to no more than one message per week.
Although we only need your email address, it would be nice if you provided a little more information to help us get to know you better!
