Meraki (now part of Cisco) Cloud Architecture Deep Dive with Sean Rhea

In this Tech Field Day presentation recorded on February 15, 2013, Sean Rhea delves into the backend architecture of Meraki’s cloud-based Dashboard. Rhea, who joined Meraki in 2007 and became the head of the backend scalability performance team in 2010, explains the tools and systems that enable Meraki to rapidly develop and deploy new features. The presentation focuses on the cloud architecture, which partitions customers into shards, each managed by a 1U server with redundancy across different data centers and providers. This setup ensures high availability and reliability, even in the event of a data center failure. The master shard acts as a demultiplexer, directing users to the appropriate shard for their data, while the system handles massive multi-tenancy with thousands of Meraki devices checking in every 45 seconds.

Rhea introduces the M-Tunnel system, which facilitates communication with Meraki devices even behind NATs, ensuring efficient data collection and configuration management. The M-Tunnel system is a resilient overlay network that reroutes traffic among shards if direct communication fails. This system, combined with the high-frequency data collection, presents significant challenges in terms of network communication and data storage. To address these, Meraki developed POTR, a high-performance system for gathering statistics. POTR uses an event-driven RPC engine with non-blocking I/O and Google protocol buffers, significantly reducing network and CPU overhead. This system allows Meraki to collect data every 45 seconds, providing high-resolution insights into network performance and issues.

The presentation also covers Little Table, a custom database designed to handle the vast amounts of data collected by Meraki devices. Little Table uses ordered trees and fixed-size buffers to efficiently store and retrieve data, minimizing disk seeks and maximizing throughput. This system allows Meraki to store detailed statistics for extended periods, enabling deep analytics and feature development. By maintaining high-resolution data, Meraki can offer detailed insights and historical analysis, which is crucial for both troubleshooting and developing new features. The presentation concludes with a discussion on the importance of data security and compliance, particularly for customers in regions with stringent data protection regulations, such as the EU.