DDN Presents at AI Field Day 9

Thinking Fast and Furious Inferencing with DDN

Watch on YouTube
Watch on Vimeo

Eric Wright, a consulting solutions architect with DDN, introduced the presentation by highlighting the growing importance of AI inference workloads, which now account for 80% of AI activity compared to training. He characterized this shift as “thinking fast and furious,” drawing an analogy to human cognitive processes, where training and inference operate in the same environment but with distinct behaviors. DDN, with decades of experience in high-performance computing (HPC) and research, aims not just to make GPUs faster, but to make them work more profitably across the entire ecosystem, addressing the inherent challenges of modern AI infrastructure.

The presentation then delved into the significant challenges in achieving productive AI, including the scarcity of GPUs and high-bandwidth memory (DRAM), as well as the increasing power demands and sustainability concerns of AI data centers. Wright explained that these constraints necessitate optimizations to maximize the efficiency of existing hardware. For training, issues like GPUs waiting on data and inefficient checkpointing are common. For inference, the KV cache is critical, and the speaker emphasized that disaggregated inference is impossible without it. Traditional storage architectures often struggle with scaling beyond a few nodes, leading to unpredictable patterns and performance bottlenecks, particularly when data spills from HBM to slower tiers like local flash, forcing constant recomputation or retrieval decisions.

DDN addresses these multifaceted problems by focusing on intelligent data management and infrastructure optimization. They aim to make flash storage act as a seamless extension to memory rather than a slow capacity tier, distinguishing their approach from conventional storage systems. By treating each memory tier distinctly and applying intelligent data placement based on reuse probability, DDN ensures that the right context is in the right place when the GPU needs it. Their architecture is designed for linear scalability and multi-tenancy, preventing single tenants from impacting overall cluster performance. This purpose-built approach, informed by their extensive HPC background, reshapes I/O to ensure optimal sequential access and maximizes parallelism, allowing flash to contribute to bandwidth and ultimately relieve the GPU, ensuring continuous awareness of data location and movement to enhance the entire inference chain.

Personnel: Eric Wright, Scott Ekstrom

To Infinia and Beyond AI Inferencing with DDN

Watch on YouTube
Watch on Vimeo

DDN, represented by Eric Wright and Scott Ekstrom, discussed their solutions for AI inferencing, building on their success in AI training. They highlighted the AI data pipeline as a continuous loop involving ingest, prepare, train, and inference stages. DDN’s Exascaler is used for the ingest, prepare, and train phases, while their Infinia product addresses inference challenges, including agentic, long context, traditional chat, and increasingly, edge inference. They described edge inference not as tiny devices, but as moderately sized local clusters in remote sites like mining operations or cruise ships, capable of running localized inference and deciding what data to send back to the core.

DDN integrates into the NVIDIA architecture, working with components like Dynamo for KV cache routing and various serving engines such as VLM and TensorRT. Their solution leverages NVIDIA’s Nixil plugin for cache offload to Infinia, presenting as high performance RAM to the NVIDIA cluster. Infinia uses a client for highly optimized communication and stores metadata with data to eliminate hops and minimize latency during lookups. DDN abstracts away the complexities of storage from developers, aiming to provide a hidden, highly performant layer that delivers an optimized experience to serving engines, thereby improving metrics like Time to First Token (TTFT) and Time Per Output Token (TPOT).

The presentation emphasized that real-world AI workloads are complex and diverse, requiring optimization across various factors including access pattern reshaping, network path awareness, zero copy data movement, parallelization, and smarter eviction strategies for KV cache. DDN focuses on understanding and improving Key Performance Indicators (KPIs) beyond raw speeds and feeds, connecting performance to business outcomes such as human user experience, tokens per watt for CFOs, and useful work per GPU hour for infrastructure teams. They aim to be trusted advisors, working with customers to architect and tune solutions that address the unique requirements and challenges of different AI applications, including multi-modal, agentic, and RAG workloads.

Personnel: Eric Wright, Scott Ekstrom

Welcome to the Loop AI Inference Lifecycle with DDN

Watch on YouTube
Watch on Vimeo

DDN, represented by Eric Wright and Scott Ekstrom, presented an overview of the complex AI inference lifecycle, highlighting their role in optimizing this process. The presentation, titled “Welcome to the Loop,” illustrated the end-to-end path, beginning with initial prompts and system instructions that form the critical context. DDN’s solution leverages its technology to load this context into an inference engine, building out the Key-Value KV cache and tokenizing the work. By integrating through a Nixto plugin and using RDMA for data movement, DDN’s Infinia acts as a distributed repository for the KV cache, making it accessible across hundreds of GPUs in a massive data center environment. This approach allows DDN to bypass local memory and NVMe, ensuring the KV cache is non-localized and useful across many GPUs, thereby addressing the challenges of scale in enterprise AI serving.

The discussion further explored the benefits of DDN’s approach for enhancing performance and efficiency, particularly for various AI workloads like retrieval augmented generation RAG and agentic flows. While initially a cold prefill for the first computation, subsequent runs achieve high KV cache reuse rates, significantly improving time to first token. DDN stores metadata alongside data for greater efficiency, especially as data scales, and emphasizes smarter eviction strategies for transient data. The company’s deep co-engineering relationship with NVIDIA and its origins in high performance computing HPC position it uniquely to tackle difficult problems in AI infrastructure, offering solutions that make GPUs work more profitably by optimizing the data path and reducing CPU involvement.

Beyond technical optimizations, the presentation touched upon the broader implications and value proposition of DDN’s solutions. They cater to a global audience, with a strong focus on ensuring data sovereignty and protection in highly regulated industries, particularly in European markets, by providing local support and dedicated infrastructure within cloud environments. DDN helps traditional storage engineers understand how their existing systems may not align with the demanding access patterns of AI inference engines. Ultimately, the company aims for fast, accurate, and cost-effective AI responses, securing data access, and supporting multi-tenancy within organizations. The profound impact of their technology is evident in diverse applications, from efficient processing of medical images like MRIs to ensuring critical infrastructure stability, emphasizing the human benefit derived from solving complex technical challenges.

Personnel: Eric Wright, Scott Ekstrom