S3 Object Storage for Real-Time Compliance Analytics at a Large US Global Bank with Scality

Ben Morge, VP of Customer Success at Scality, presented a deployment of Scality RING for a large US bank that needed to store 40 petabytes of Splunk SmartStore data across two sites.  The bank required active-active replication and a one-year data retention period. RING’s S3 compatibility enabled seamless integration with Splunk, allowing the indexers to tier data from hot, fast flash storage to the warm Scality RING, which consisted of 80 servers.  The data is immutable on the ring for one year, with the Splunk application handling data deletion.

The Scality deployment leverages S3 for data storage. The solution employs a two-site architecture, where each site features Splunk indexers and a hot storage cluster. The indexers are responsible for replicating indexes and references to objects stored on the ring. Scality manages object replication, utilizing cross-region replication, which is the S3 standard. The system addresses network issues by employing infinite replay to ensure reliable replication and decoupling storage from compute.

The presentation highlighted a couple of challenges and their solutions. The initial ingest exceeding firewall throughput was resolved with infinite replay. The initial difficulties with the read traffic exceeding CPU capabilities, leading to overwhelmed flash caches, were addressed by decoupling the architecture and adding compute resources to handle the metadata layer and S3 stateless services, resulting in simultaneous 75 gigabytes throughput on both sites and a fully replicated cluster with all the objects replicated in under two minutes.  The customer’s active production has been running successfully for over five years.