Concurrency at NVIDIA Cloud Partner (NCP) Scale with DDN

Eric Wright and Scott Ekstrom from DDN presented on achieving concurrency at NVIDIA Cloud Partner (NCP) scale, focusing on optimizing AI infrastructure to keep GPUs productive and profitable. They highlighted that AI has transformed storage from a perceived “boring” component into a complex challenge, as different AI models have vastly varying data consumption patterns. DDN aims to maximize value from AI investments by addressing these intricacies.

The presentation delved into three distinct model patterns: hybrid models like Nemotron 3 (using Mamba attention), coding agents (large parameter models with fan-out effects and tool calls), and traditional reasoning models. Each pattern exhibits unique KV cache behaviors and data access needs, yet they all rely on the same underlying GPUs, network, and storage. Eric illustrated the rapid growth of context windows, with a 128K context translating to 1 gigabyte of KV cache, and noted that model weights themselves can consume significant GPU memory (e.g., 141GB for an H200 before a single prompt). This necessitates sophisticated management of data residency and intelligent caching strategies for optimal performance.

DDN’s approach focuses on a holistic system view, rather than isolated optimizations, recognizing that changes in one area impact the entire infrastructure. Their engineering team is dedicated to designing and consuming resources efficiently, optimizing data flow with techniques like zero-copy RDMA to get data off the CPU path and ensuring shared storage access across entire clusters. The goal is to create an adaptive platform, like their Infinia solution, that can identify and solve performance bottlenecks across different phases of AI workloads, from S3 object storage for enterprise data and training, to POSIX file systems, and the critical low-latency KV cache for GPU computations. This requires robust observability to understand how workloads behave and to continuously optimize for efficiency, cost, and security.