Watch on YouTube
Watch on Vimeo
This presentation introduces MinIO MemKV, a critical new memory layer designed to address the challenges of scaling AI inference workloads, particularly for agentic applications. The core problem stems from the increasing size of context memory, known as the KV cache, which frequently exceeds the high-bandwidth memory (HBM) capacity of GPUs. In agentic workloads, where requests build upon previous interactions, this context memory constantly expands. When HBM is exhausted, GPUs resort to evicting and recomputing the KV cache, leading to wasted cycles during the “pre-fill” phase and significantly increasing the “Time To First Token” (TTFT) for users. This inefficient utilization plagues modern inference deployments, where GPUs might appear 100% utilized but are largely performing redundant computations.
MinIO MemKV offers a purpose-built solution by creating a distributed shared memory layer that GPUs can access quickly. It bypasses traditional file systems and kernel overhead by leveraging direct NVMe access to achieve microsecond latency. Unlike conventional enterprise storage, MemKV is engineered as an extension of memory, free from the “baggage” of durability features unnecessary for transient KV cache data. Benchmarks demonstrate impressive gains: a single H200 GPU node with two MemKV servers can handle up to 43 times more concurrent requests, achieve aggregated throughputs of nearly 97 gigabytes per second, and scale linearly to support vast superpods with thousands of GPUs and petabytes of context memory, delivering 16,000 concurrent requests per second and 12 terabits per second throughput. This dramatically reduces TTFT and ensures that GPUs focus on useful decoding rather than repetitive recomputation.
The speakers highlight that this bottleneck is fundamentally a “software problem” best addressed through optimized software, rather than complex hardware solutions like CXL, which is deemed too low-level and slow to adapt. MemKV’s approach allows for a “G3.5” memory tier that efficiently serves large KV cache blocks (2MB to 64MB tensors) directly from NVMe, avoiding file system metadata overhead. This enables superior effective GPU utilization, leading to significant cost reductions, estimated at over $2 million per year for a single H200 node in the public cloud. By providing a fast, scalable, and shared context memory, MemKV ensures GPUs perform meaningful work, boosting efficiency and handling the high concurrency demanded by modern AI inference.
Personnel: AB Periasamy, Dil Radhakrishnan
Thank you for being part of the Tech Field Day community! Our mailing list is a great way to stay up to date on our events and technical content, and we appreciate your signup.
We promise that we’ll never spam you, send ads, or sell your information. This list will only be used to communicate with our community about our events and content. And we’ll limit it to no more than one message per week.
Although we only need your email address, it would be nice if you provided a little more information to help us get to know you better!
