To Infinia and Beyond AI Inferencing with DDN

DDN, represented by Eric Wright and Scott Ekstrom, discussed their solutions for AI inferencing, building on their success in AI training. They highlighted the AI data pipeline as a continuous loop involving ingest, prepare, train, and inference stages. DDN’s Exascaler is used for the ingest, prepare, and train phases, while their Infinia product addresses inference challenges, including agentic, long context, traditional chat, and increasingly, edge inference. They described edge inference not as tiny devices, but as moderately sized local clusters in remote sites like mining operations or cruise ships, capable of running localized inference and deciding what data to send back to the core.

DDN integrates into the NVIDIA architecture, working with components like Dynamo for KV cache routing and various serving engines such as VLM and TensorRT. Their solution leverages NVIDIA’s Nixil plugin for cache offload to Infinia, presenting as high performance RAM to the NVIDIA cluster. Infinia uses a client for highly optimized communication and stores metadata with data to eliminate hops and minimize latency during lookups. DDN abstracts away the complexities of storage from developers, aiming to provide a hidden, highly performant layer that delivers an optimized experience to serving engines, thereby improving metrics like Time to First Token (TTFT) and Time Per Output Token (TPOT).

The presentation emphasized that real-world AI workloads are complex and diverse, requiring optimization across various factors including access pattern reshaping, network path awareness, zero copy data movement, parallelization, and smarter eviction strategies for KV cache. DDN focuses on understanding and improving Key Performance Indicators (KPIs) beyond raw speeds and feeds, connecting performance to business outcomes such as human user experience, tokens per watt for CFOs, and useful work per GPU hour for infrastructure teams. They aim to be trusted advisors, working with customers to architect and tune solutions that address the unique requirements and challenges of different AI applications, including multi-modal, agentic, and RAG workloads.