Accelerating GPUs with CAIOS, LOTA, and DFS on Coreweave Storage

Chris Milsted, a specialist field engineer, explained that CoreWeave’s primary goal is to keep expensive GPUs as active and close to 100% utilized as possible, especially as modern GPUs like the H100 and Vera Ruben deliver rapidly increasing bandwidth. These systems demand immense data speeds, with front-end network connections around 200 gigabits, posing a significant challenge for efficient data delivery. The presentation aims to demonstrate how CAIOS, LOTA (Local Object Transport Accelerator), and DFS (Distributed File System) provide a significant advantage in meeting these demands.

LOTA leverages local NVMe disks within each GPU node (e.g., eight 7.62 TB NVMe drives, with six dedicated for cache, providing 40-41 TB per node) to transparently cache data. When a user or workload accesses an object for the first time, it is pulled from the backend CoreWeave AI Object Storage into the local cache. Subsequent accesses are then served directly from this high-speed local cache, drastically reducing latency. While currently a read-only cache, CoreWeave also offers cross-region write acceleration, allowing local writes to be buffered to a local object storage array and then asynchronously destaged to a remote, durable object storage, addressing challenges for multi-region deployments or large checkpoint files.

This technology solves critical problems such as accelerating access to massive datasets, which can span petabytes or exabytes, and enabling fast data access across different data centers without needing to create costly duplicate copies of entire datasets. LOTA operates transparently, meaning users only need to use a different endpoint without needing to manage the cache directly. A demonstration showcased significant improvements in time to first byte when data was served from the local cache compared to pulling it across transatlantic links from remote object storage. Additionally, CoreWeave’s MCP offers easy monitoring and debugging of cache performance, while DFS provides a low-latency alternative for file system protocols.