Taming Unstructured Data Orchestration with Hammerspace

In this session, Floyd Christofferson and Chad Smith from Hammerspace will step through the key capabilities of Hammerspace Global Data Environment software, and how it automates unstructured data orchestration across multi-vendor, multi-site, and often multi-cloud storage environments. It will focus specifically on solutions to the problems of when data that is needed for AI pipelines is distributed across silos, sites, and clouds.

Christofferson and Smith discuss the capabilities of Hammerspace’s Global Data Environment software for automating unstructured data orchestration across various storage environments, including multi-vendor, multi-site, and multi-cloud infrastructures. They focus on how this can be particularly beneficial for AI workflows, where data is often distributed across different locations and silos.

Hammerspace’s solution involves separating file system metadata from the actual data, elevating it above the infrastructure layer into a global metadata control plane. This allows for a common view of files across different storage systems and locations, enabling transparent and automated data orchestration without disrupting user access or requiring data movement.

The software is Linux-based and includes two components: Anvil servers for metadata control and DSX nodes for I/O handling. It supports multi-protocol access, including NFS, parallel NFS, and S3, and allows for the setting of objective-based policies for data management, including protection, tiering, and geographical considerations.

Hammerspace can be installed on various platforms, including bare metal, cloud instances, and VMs, and it facilitates seamless integration of on-premises storage with cloud resources. This enables use cases like bursting AI workloads to the cloud, managing data across global sites, and optimizing compute resource costs by automating data movement to the most cost-effective locations.

Floyd provides examples of Hammerspace’s application in different industries, such as online gaming, rocket company Blue Origin, and a data center in London that saves costs by orchestrating render jobs to cheaper cloud regions.