What’s Our Vector, Victor: A Deep Dive into Vector Databases and KV Cache with DDN

Scott Ekstrom and Eric Wright from DDN presented on vector databases and KV cache, emphasizing how tokens from large language model prompts are vectorized through an embedding table, creating learned vectors. These vectors, capturing context and semantics across multiple model layers, are stored in the KV cache, leading to significant data sizes even for small prompts. While both involve vectors, the presentation clarified that KV cache vectors are dynamic, constantly changing with queries and neighboring data, making direct comparisons or alignments with static vector database entries challenging due to their evolving nature. Context is crucial for vector databases to avoid ambiguity, as a single word without surrounding information can have multiple meanings and become useless.

DDN positions itself as a critical enabler for scaling vector database environments. Traditionally, vector databases reside in expensive DRAM, posing a significant challenge when scaling to millions or billions of vectors. DDN’s approach, leveraging solutions like Milvus and advancements like AISAQ, moves these vector indexes from local memory to shared, persistent block storage accessible via RDMA. This migration drastically reduces DRAM usage (e.g., a 3200x reduction in memory for Milvus in testing) and allows for independent scaling of compute and capacity, maximizing GPU productivity. While this may introduce a small latency trade-off, it offers substantial cost savings and enables efficient semantic searches with high recall rates, such as 9.8 out of 10 nearest neighbors in large datasets.

The presentation also highlighted the pivotal role of vector databases in Retrieval Augmented Generation (RAG) systems. These systems rely on vector databases to provide grounded, current, and permissioned “sources of truth” for LLMs, preventing hallucinations and ensuring relevant responses. Managing these vector databases at scale requires robust database functionalities beyond simple indexing, including live updates, hybrid search capabilities, metadata filtering, and comprehensive operational support. DDN’s Infinia storage platform is engineered to handle the high volume of small, rapid reads characteristic of these workloads, providing linear scalability, cost-effectiveness by utilizing NVMe, and maintaining tight performance metrics (P95/P99 latencies). Ultimately, DDN sees storage becoming the central plumbing of the NVIDIA AI Data Platform, adapting to new architectures and enabling expansive, reliable AI applications.