Watch on YouTube
Watch on Vimeo
This presentation detailed Solidigm’s collaboration with Metrum to explore how storage, specifically Solidigm’s SSDs, can effectively function as AI memory for Retrieval Augmented Generation (RAG) and Key-Value (KV) cache operations. Metrum’s core challenge involved ingesting vast amounts of video, generating detailed metadata from it, and creating a quickly searchable database for AI inference. The objective was to determine if SSDs could supplant or improve upon traditional memory-based vector databases, thereby optimizing performance and reducing costs. This led to an evaluation comparing HNSW, a primarily memory-based vector database, with DiskANN, a solution explicitly optimized for storage, using Solidigm SSDs.
Metrum’s benchmarks across 1 million, 10 million, and 100 million datasets revealed compelling results. While HNSW showed a slight edge at lower concurrencies with smaller datasets, DiskANN demonstrated superior performance as concurrency increased, especially with real-world video pipeline data. This indicated that a purpose-built, storage-first approach could outperform memory-centric solutions in critical scenarios, proving that significant cost savings on DRAM could be achieved without sacrificing performance for large-scale vector databases. The ability to move these substantial datasets from expensive DRAM to more economical SSDs offers a powerful advantage for AI infrastructure.
Beyond vector database lookups, the presentation emphasized the critical role of SSDs in KV-cache offload. By caching recurrent queries on Solidigm drives, the system drastically improved the “time to first token” and eliminated redundant GPU recomputation, which otherwise wastes power and delays responses. This efficiency is crucial for serving hundreds or thousands of concurrent users, providing faster, more responsive AI interactions, and freeing up valuable GPU compute cycles for other tasks. This concept extends beyond video and applies to any large-scale AI system that benefits from efficiently persistent context and reduced re-inference, highlighting how intelligent storage integration can lead to substantial gains in both performance and operational efficiency across the entire AI pipeline.
Personnel: Allyn Malventano
Thank you for being part of the Tech Field Day community! Our mailing list is a great way to stay up to date on our events and technical content, and we appreciate your signup.
We promise that we’ll never spam you, send ads, or sell your information. This list will only be used to communicate with our community about our events and content. And we’ll limit it to no more than one message per week.
Although we only need your email address, it would be nice if you provided a little more information to help us get to know you better!
