What can we optimize in AI inference that meaningfully increases productive throughput? How can we lower the cost per token once a model is in production? DDN goes past the “what if?” to field-proven deployments and what our architects learned in customer environments. The team will share techniques and trade-offs that lead to more productive AI with real use cases in Enterprise AI and NCPs.
Watch on YouTube
Watch on Vimeo
Eric Wright and Scott Ekstrom from DDN emphasize that DDN’s core goal is to ensure GPUs operate productively and profitably, allowing customers to extract maximum value from their AI data investments. They approach AI infrastructure with a “first principles” mindset, focusing on understanding actual business outcomes rather than just hero metrics. This involves deep dives into data skills, ML Ops, governance, and analytics to ensure systems are built to meet evolving needs for enterprise and sovereign customers, as well as service partners.
The presentation outlines key problems DDN solves, particularly related to GPU underutilization in training and checkpointing due to slow data movement. For shared clusters, they address the complexities of varied tenant profiles, priorities, and consumption patterns, which can lead to unpredictable job performance and nonlinear expansion costs. In inference, a critical focus is optimizing the KV (Key-Value) cache to avoid costly context recomputation. DDN employs two distinct data engines: Exascaler, which is robust for ingestion, data preparation, training, and fine-tuning, leveraging its HPC origins; and Infinia, specifically designed for inference, particularly for agentic and long-context edge inference, with a strong emphasis on KV cache optimization.
DDN’s strategy for performance and profitability centers on a shared KV cache tier, ensuring cross-node and cross-cluster cache coherency. They cite a case where using Infinia for KV cache reduced a critical turn time from nearly 32 seconds to under 4 seconds. The adaptive AI infrastructure supports optimized inference workflows by splitting pre-fill and decode operations, using zero-copy RDMA for efficient KV block movement between GPUs without CPU involvement. Crucially, multi-tenancy is built into the KV cache layer, ensuring isolation and security by attaching tenant keys to data blocks. This flexible architecture allows for dynamic allocation of resources, such as creating distinct embedding and generation pools, enabling efficient and scalable AI operations across various workloads.
Personnel: Eric Wright, Scott Ekstrom
Watch on YouTube
Watch on Vimeo
Eric Wright and Scott Ekstrom from DDN presented on achieving concurrency at NVIDIA Cloud Partner (NCP) scale, focusing on optimizing AI infrastructure to keep GPUs productive and profitable. They highlighted that AI has transformed storage from a perceived “boring” component into a complex challenge, as different AI models have vastly varying data consumption patterns. DDN aims to maximize value from AI investments by addressing these intricacies.
The presentation delved into three distinct model patterns: hybrid models like Nemotron 3 (using Mamba attention), coding agents (large parameter models with fan-out effects and tool calls), and traditional reasoning models. Each pattern exhibits unique KV cache behaviors and data access needs, yet they all rely on the same underlying GPUs, network, and storage. Eric illustrated the rapid growth of context windows, with a 128K context translating to 1 gigabyte of KV cache, and noted that model weights themselves can consume significant GPU memory (e.g., 141GB for an H200 before a single prompt). This necessitates sophisticated management of data residency and intelligent caching strategies for optimal performance.
DDN’s approach focuses on a holistic system view, rather than isolated optimizations, recognizing that changes in one area impact the entire infrastructure. Their engineering team is dedicated to designing and consuming resources efficiently, optimizing data flow with techniques like zero-copy RDMA to get data off the CPU path and ensuring shared storage access across entire clusters. The goal is to create an adaptive platform, like their Infinia solution, that can identify and solve performance bottlenecks across different phases of AI workloads, from S3 object storage for enterprise data and training, to POSIX file systems, and the critical low-latency KV cache for GPU computations. This requires robust observability to understand how workloads behave and to continuously optimize for efficiency, cost, and security.
Personnel: Eric Wright, Scott Ekstrom
Watch on YouTube
Watch on Vimeo
Eric Wright and Scott Ekstrom of DDN presented on the role of KV cache within the data path, acknowledging that their title might be misleading, as the core focus is on measuring what truly matters in a complex, multivariate optimization problem. They emphasized that system changes are not isolated, likening it to squeezing a balloon where pressure shifts elsewhere. The presentation highlights the importance of observing key performance indicators (KPIs) to understand the cascading effects of adjustments in an AI infrastructure, particularly as various metrics, such as CPU, memory, and networking, are interconnected and influence overall system performance.
The presentation demonstrated the significant impact of offloading KV cache, showing a reduction in time to first token (TTFT) from 6 seconds to 370 milliseconds in a small, CPU-only inference environment. This efficiency gain, however, introduces trade-offs, such as increased S3 requests, creating new operational considerations. A critical challenge addressed is the loss of cache upon node restarts, which necessitates rebuilding the cache from scratch, a process that can take 6 seconds. DDN aims to drastically cut this recovery time to 560 milliseconds by enabling rapid loading from the KV cache, acknowledging that while individual node restarts may not be frequent, model reloading and multiple competing models make this a meaningful impact for their customers.
DDN’s approach extends beyond just KV cache optimization, focusing on an enterprise data platform that supports both training and inference. This includes managing “dark data” to ensure rapid classification and availability of relevant data for real-time conversations, efficient chunking and embedding, and robust governance with multi-tenancy and granular permissions. The goal is to provide accurate, semantic, and searchable content efficiently, adaptable to changing models and consumption patterns. By building on a strong foundation, DDN seeks to provide deep insights into customer environments and help organizations meet board-level metrics for AI adoption by ensuring speed, efficiency, security, and adaptability in the face of constantly evolving AI technologies.
Personnel: Eric Wright, Scott Ekstrom
Watch on YouTube
Watch on Vimeo
Scott Ekstrom and Eric Wright from DDN presented on vector databases and KV cache, emphasizing how tokens from large language model prompts are vectorized through an embedding table, creating learned vectors. These vectors, capturing context and semantics across multiple model layers, are stored in the KV cache, leading to significant data sizes even for small prompts. While both involve vectors, the presentation clarified that KV cache vectors are dynamic, constantly changing with queries and neighboring data, making direct comparisons or alignments with static vector database entries challenging due to their evolving nature. Context is crucial for vector databases to avoid ambiguity, as a single word without surrounding information can have multiple meanings and become useless.
DDN positions itself as a critical enabler for scaling vector database environments. Traditionally, vector databases reside in expensive DRAM, posing a significant challenge when scaling to millions or billions of vectors. DDN’s approach, leveraging solutions like Milvus and advancements like AISAQ, moves these vector indexes from local memory to shared, persistent block storage accessible via RDMA. This migration drastically reduces DRAM usage (e.g., a 3200x reduction in memory for Milvus in testing) and allows for independent scaling of compute and capacity, maximizing GPU productivity. While this may introduce a small latency trade-off, it offers substantial cost savings and enables efficient semantic searches with high recall rates, such as 9.8 out of 10 nearest neighbors in large datasets.
The presentation also highlighted the pivotal role of vector databases in Retrieval Augmented Generation (RAG) systems. These systems rely on vector databases to provide grounded, current, and permissioned “sources of truth” for LLMs, preventing hallucinations and ensuring relevant responses. Managing these vector databases at scale requires robust database functionalities beyond simple indexing, including live updates, hybrid search capabilities, metadata filtering, and comprehensive operational support. DDN’s Infinia storage platform is engineered to handle the high volume of small, rapid reads characteristic of these workloads, providing linear scalability, cost-effectiveness by utilizing NVMe, and maintaining tight performance metrics (P95/P99 latencies). Ultimately, DDN sees storage becoming the central plumbing of the NVIDIA AI Data Platform, adapting to new architectures and enabling expansive, reliable AI applications.
Personnel: Eric Wright, Scott Ekstrom
Thank you for being part of the Tech Field Day community! Our mailing list is a great way to stay up to date on our events and technical content, and we appreciate your signup.
We promise that we’ll never spam you, send ads, or sell your information. This list will only be used to communicate with our community about our events and content. And we’ll limit it to no more than one message per week.
Although we only need your email address, it would be nice if you provided a little more information to help us get to know you better!
