Ben Jarvis, Lead Storage Architect, discusses the innovative architecture of the Quantum Myriad all-flash scale-out storage system. Myriad was designed from the ground up to handle high-performance video and HPC workloads, leveraging large scale-out internet-facing service paradigms. It utilizes a microservices container-based system running on Kubernetes, deployed on white box commodity hardware, and is optimized for NVMe flash storage and high-performance RDMA Ethernet networks.
The system is designed to invert traditional storage models by optimizing software to fully utilize the hardware’s capabilities, focusing on memory bandwidth and minimizing context switches. Myriad’s architecture supports inline data protection, data reduction through compression and deduplication, and presents a self-contained cluster to customers.
Jarvis explains the integration with Kubernetes for state management and the system’s approach to handling physical storage on nodes. Myriad operates on a transactional key-value store basis, supporting transactions necessary for a POSIX-compliant file system. It employs a redirected write system with data protection through mirroring and erasure encoding, optimizing for low-latency, high-concurrency operations. The system’s design avoids performance bottlenecks common in traditional storage systems by adopting a work-at-risk approach for managing transactions. This method allows for independent operations with synchronization only at the end, significantly reducing the probability of conflicts and enabling the system to scale efficiently. Myriad is optimized for workloads that do not involve simultaneous writes to the same file by multiple nodes, steering clear of legacy HPC workloads that do not align with its architectural advantages.
Personnel: Ben Jarvis
Thank you for being part of the Tech Field Day community! Our mailing list is a great way to stay up to date on our events and technical content, and we appreciate your signup.
We promise that we’ll never spam you, send ads, or sell your information. This list will only be used to communicate with our community about our events and content. And we’ll limit it to no more than one message per week.
Although we only need your email address, it would be nice if you provided a little more information to help us get to know you better!
