Thinking Fast and Furious Inferencing with DDN

Eric Wright, a consulting solutions architect with DDN, introduced the presentation by highlighting the growing importance of AI inference workloads, which now account for 80% of AI activity compared to training. He characterized this shift as “thinking fast and furious,” drawing an analogy to human cognitive processes, where training and inference operate in the same environment but with distinct behaviors. DDN, with decades of experience in high-performance computing (HPC) and research, aims not just to make GPUs faster, but to make them work more profitably across the entire ecosystem, addressing the inherent challenges of modern AI infrastructure.

The presentation then delved into the significant challenges in achieving productive AI, including the scarcity of GPUs and high-bandwidth memory (DRAM), as well as the increasing power demands and sustainability concerns of AI data centers. Wright explained that these constraints necessitate optimizations to maximize the efficiency of existing hardware. For training, issues like GPUs waiting on data and inefficient checkpointing are common. For inference, the KV cache is critical, and the speaker emphasized that disaggregated inference is impossible without it. Traditional storage architectures often struggle with scaling beyond a few nodes, leading to unpredictable patterns and performance bottlenecks, particularly when data spills from HBM to slower tiers like local flash, forcing constant recomputation or retrieval decisions.

DDN addresses these multifaceted problems by focusing on intelligent data management and infrastructure optimization. They aim to make flash storage act as a seamless extension to memory rather than a slow capacity tier, distinguishing their approach from conventional storage systems. By treating each memory tier distinctly and applying intelligent data placement based on reuse probability, DDN ensures that the right context is in the right place when the GPU needs it. Their architecture is designed for linear scalability and multi-tenancy, preventing single tenants from impacting overall cluster performance. This purpose-built approach, informed by their extensive HPC background, reshapes I/O to ensure optimal sequential access and maximizes parallelism, allowing flash to contribute to bandwidth and ultimately relieve the GPU, ensuring continuous awareness of data location and movement to enhance the entire inference chain.