Where Does KV Cache Fit in the Whole Data Path with DDN

Eric Wright and Scott Ekstrom of DDN presented on the role of KV cache within the data path, acknowledging that their title might be misleading, as the core focus is on measuring what truly matters in a complex, multivariate optimization problem. They emphasized that system changes are not isolated, likening it to squeezing a balloon where pressure shifts elsewhere. The presentation highlights the importance of observing key performance indicators (KPIs) to understand the cascading effects of adjustments in an AI infrastructure, particularly as various metrics, such as CPU, memory, and networking, are interconnected and influence overall system performance.

The presentation demonstrated the significant impact of offloading KV cache, showing a reduction in time to first token (TTFT) from 6 seconds to 370 milliseconds in a small, CPU-only inference environment. This efficiency gain, however, introduces trade-offs, such as increased S3 requests, creating new operational considerations. A critical challenge addressed is the loss of cache upon node restarts, which necessitates rebuilding the cache from scratch, a process that can take 6 seconds. DDN aims to drastically cut this recovery time to 560 milliseconds by enabling rapid loading from the KV cache, acknowledging that while individual node restarts may not be frequent, model reloading and multiple competing models make this a meaningful impact for their customers.

DDN’s approach extends beyond just KV cache optimization, focusing on an enterprise data platform that supports both training and inference. This includes managing “dark data” to ensure rapid classification and availability of relevant data for real-time conversations, efficient chunking and embedding, and robust governance with multi-tenancy and granular permissions. The goal is to provide accurate, semantic, and searchable content efficiently, adaptable to changing models and consumption patterns. By building on a strong foundation, DDN seeks to provide deep insights into customer environments and help organizations meet board-level metrics for AI adoption by ensuring speed, efficiency, security, and adaptability in the face of constantly evolving AI technologies.