Hammerspace Presents at AI Field Day 8

Unstructured Data and the New Economics of Enterprise AI with Hammerspace

Watch on YouTube
Watch on Vimeo

Hammerspace operates as a non-proprietary, standards-based, software-defined data platform designed to unify an enterprise’s unstructured data estate, encompassing object storage and NAS, without requiring data movement. It orchestrates and automates data movement based on policies, enabling acceleration to GPU processors via its PNFS parallel file system, which leverages standard NFS for easy integration without custom clients. The presentation highlighted that while enterprises need comprehensive visibility into their data for AI, traditional centralization methods are no longer viable due to the sheer volume and distribution of data. Current challenges for scaling AI applications include severe constraints on power, compute (GPUs), and even storage components like SSDs and disk drives, with cloud providers also facing capacity limitations. These realities mean simply buying more hardware is often not an option, exposing the limitations of a centralized, siloed data approach.

Hammerspace addresses these constraints by maximizing and optimizing existing resources. This includes unifying stranded capacity within distributed NAS infrastructures and incorporating local NVMe storage (termed “Tier Zero”) on AI servers into a global namespace for high-performance, low-latency access. It modernizes current commodity storage by enabling a parallel file system to run on top, accelerating data to GPU clusters while continuing to utilize existing hardware. Several customer case studies illustrate this approach: a semiconductor company achieved better performance for LLM development by leveraging existing storage with PNFS; another customer complemented their HPC storage by utilizing Tier Zero for high-performance local storage, saving costs; and a company shifted from NAS to object storage for data scientists, extending their namespace to GCP for additional GPU capacity, all transparently. Hammerspace also supports multi-datacenter and multi-cloud strategies, allowing customers to dynamically scale GPU workloads across Azure, Nebul, and other neo-clouds, abstracting the underlying infrastructure from users.

The core message is that growth in AI is a strategic imperative, but traditional resource expansion is currently unsustainable. Hammerspace enables agility by facilitating the consumption of new resources, whether on-premises or across various cloud providers, as they become available, without necessitating complete system re-architecture. It also integrates data sovereignty capabilities through intelligent orchestration rules, ensuring data remains within defined geographic or regulatory boundaries. While the solution is primarily aimed at large enterprises grappling with petabyte-scale, performance-intensive AI and HPC problems within sophisticated environments, Hammerspace continuously strives to simplify its interface and abstract complexity, making advanced data management more accessible for these demanding workloads. The goal is to empower organizations to continue innovating and growing their AI capabilities despite current hardware and infrastructure constraints by optimizing the data layer.

Personnel: Kurt Kuckein

Delivering Agility in Partner Environments with Hammerspace and Hitachi Vantara

Watch on YouTube
Watch on Vimeo

Hammerspace and Hitachi Vantara have forged a strong partnership to deliver agility in partner environments, particularly for AI workloads. Hitachi Vantara’s Chief AI Strategist, David Chapa, explained that this collaboration is not competitive but rather complementary. Hammerspace does not aim to displace existing storage infrastructure but instead provides a layer that enhances current storage investments. This synergistic relationship, which began two years prior, focuses on integrating their respective strengths to provide comprehensive solutions for the burgeoning AI market, managed under Hitachi’s IQ portfolio.

A core aspect of this partnership lies in their distinct yet interconnected roles. Hitachi Vantara positions Hammerspace as the control plane for information, granting customers crucial visibility into their vast data estates to understand data location and optimize its use for AI. Concurrently, Hitachi Vantara’s VSP1 storage serves as the foundational control plane for the data itself, known for its resilience, reliability, and guarantees for uptime and data efficiency. By combining VSP1’s robust block and object storage with Hammerspace’s file-level capabilities, the partnership offers a complete solution that enables intelligent data management for diverse AI ecosystems, including HPC clusters and GPU farms.

This collaboration allows customers to bridge the gaps between various storage types—file, object, and block—within complex, large-scale environments. The integrated solution simplifies management and offers cost-effective scalability for AI initiatives. Use cases range from hybrid multi-cloud deployments to critical AI workflows like fraud detection in financial services and operational analysis in law enforcement, as well as traditional HPC workloads in research. Both companies leverage their significant domain expertise to provide consultative guidance, helping organizations transition AI projects from proof of concept to scalable production. They highlight that success in AI hinges on effective data movement, locality, and memory management rather than solely on compute power, optimizing resource utilization for their clients.

Personnel: David Chapa, Kurt Kuckein