Leading Space Agency’s Long-Term Scientific Storage at Scale with Scality

Scality’s presentation at Cloud Field Day 23 focused on their collaboration with a major European space agency, which utilizes Scality RING to manage massive scientific datasets in a hybrid storage model. The agency, dealing with data from approximately 200 satellites, faced challenges with legacy storage solutions and the need for a cost-effective, scalable, and easily accessible system for both live and archival data. Scality’s solution utilizes S3 as a unified namespace, providing a single access point for data regardless of its location, whether on hot or cold storage.

The solution employs a multi-tiered approach, where live data is stored on the RING for active analysis and subsequently moved to tape for long-term archiving after six months. This vast amount of cold data, representing hundreds of petabytes, is managed using a custom-built API called TLP (Tape Library Protocol) to integrate with HSMs from partners such as HP, Atempo, IBM, and Spectra. TLP handles the retrieval and storage of data to tape, providing transparent access for users through the S3 Glacier API. This provides cost savings and energy efficiency by moving data to tape when it is not frequently accessed.

This architecture offers several advantages, including data durability through a three-site stretch ring, with data replicated across two tape libraries for enhanced resilience. The agency’s users and applications interact with the data via a single namespace using S3, unaware of the underlying complexity of the hybrid storage system. This transparency, combined with the cost-effectiveness of the solution and security features like object lock, has made Scality’s solution a key enabler for long-term data access and efficient workflows for the space agency.