Taming Data Estate Chaos for AI with Hammerspace

Hammerspace introduces itself as a “data company,” distinguishing itself from traditional storage vendors by offering a solution that addresses the complex data demands of modern infrastructure, particularly for AI workloads. The core concept behind Hammerspace is an instantly accessible, infinite virtual space that disaggregates data from its underlying infrastructure, enabling it to reside in any location, across any cloud, and on any storage backend, thereby eliminating data silos. This is achieved by assimilating metadata from existing storage systems into a single, global namespace, managed by metadata servers outside the data path. This approach not only accelerates data pipelines but also enhances existing infrastructure and enables rapid, easy integration of new technologies, providing users with visibility and access to all their data within minutes, rather than requiring lengthy, costly data migrations.

Hammerspace extends its capabilities to address critical challenges in AI infrastructure, including the current tight market and rising flash memory costs. The solution leverages underutilized flash storage within existing environments by aggregating systems and intelligently orchestrating data placement across tiers. It introduces “Tier Zero,” which consumes and aggregates local flash within compute (CPU and GPU) clusters into the global namespace, providing extremely high-performance storage by eliminating network latency. Hammerspace also treats cloud storage as a direct extension of on-premises infrastructure, not just a destination for data, thereby maximizing the use of available flash resources. The software-defined platform ensures data portability and access through a parallel file system (PNFS v4.2) and multi-protocol access (S3, NFS, SMB). Importantly, its policy-driven orchestration automates data movement and ensures data durability and availability through redundant metadata nodes and erasure coding across storage systems. It also centralizes privileged access and security policies, allowing permissions to follow data regardless of its physical location, critical for cross-border data compliance and auditability, and supports rich custom metadata beyond basic POSIX attributes.

Customer examples illustrate these benefits, such as a digital payments company that reduced storage costs by $5 million and simplified workflows for 3,000 data scientists by providing parallel file system access over object storage and enabling hybrid cloud agility. Another customer, facing a 3-4x increase in performance demand from new NVIDIA servers, leveraged Hammerspace to maintain existing NAS systems while deploying high-performance NVMe storage, avoiding significant new infrastructure investments. For inference workloads where latency is critical, Hammerspace can use policies to preload entire projects into local NVMe (Tier Zero) directly connected to GPUs, maintaining high performance and data consistency across globally distributed inference farms. Ultimately, through its integration with platforms like the NVIDIA AI Data Platform, Hammerspace goes beyond merely unifying data access; it truly unlocks the value within data by automating data preparation and orchestration, moving organizations from data chaos to a state of AI-ready data, often allowing interaction with the system via natural language for streamlined management.