AI CPU Deepdive, Sandboxes, and security demo with CoreWeave

Deok Filho, Product Manager for Sandbox at CoreWeave, discusses the emergence of sandboxes as a crucial new compute primitive for running AI agents. He highlights that while containers have been used for a long time with controls over networking, storage, and identity, agents, which have become proficient in complex tasks like booking tickets or writing and deploying code, require a new level of containment. This necessity arises from agents performing “unintentional things,” leading to security concerns and the need for a compute primitive that offers more granular control than traditional serverless container solutions like AWS Lambda or Google Cloud Run, which are not designed for the dynamic, often untrusted code generated by agents.

The core difference in a sandbox, as offered by CoreWeave, is its ability to provide extremely granular control over an agent’s environment. Unlike serverless functions that run repetitive, known code, agents generate novel, untrusted code that requires robust isolation and specific governance. CoreWeave employs a Kata-based microVM container runtime, not OpenShell, to achieve this, building granular controls for networking, storage, and identity on top. This allows for precise restrictions, such as limiting an agent’s access to specific API endpoints or CIDR ranges. Sandboxes primarily utilize CPUs for both training (especially reinforcement learning environments) and production inference, addressing an unforeseen surge in demand for CPU capacity. Key use cases include coding agents, autonomous agents for QA or cybersecurity testing, RL environments for model post-training, and CI/CD, where agents produce high volumes of code.

CoreWeave’s solution addresses customer needs for useful capacity, reliability, and reduced operational overhead. They leverage existing CPU cores on GPU nodes (which often sit idle during GPU-heavy training) to run sandboxes, maximizing utilization of reserved compute. For bursty agent workloads, CoreWeave has developed a multi-zonal capacity management system. This system intelligently fills a customer’s reserved nodes first, then utilizes spot nodes, and finally spills over to CoreWeave’s serverless offering for peak demand, ensuring continuous service while managing costs. Features include fast snapshot and restore for agent trajectories, persistent distributed file systems for agent context and memory, and network rules with SNI filtering to enforce guardrails across a cluster. CoreWeave acknowledges that fine-grained security features like per-action policy enforcement or secrets injection often require external AI gateways, although they are building some of these capabilities into their own platform and plan to offer GPU sandboxes soon.