Watch on YouTube
Watch on Vimeo
Rob Bird makes the case for simulation with a chess game. In Paul Morphy’s 1850s Opera Game, Morphy faced a position with about 46 legal moves, 40 of which would have weakened his own position, and he found a queen sacrifice that forced checkmate in two moves. Bird argued that every infrastructure change is a move on a far more complicated board, where the components of an environment are the pieces, their dependencies are the possible moves, and each change can set off a chain of reactions across other systems. He said a modest AWS test environment with a few dozen load balancers, containers, and databases has about 10 to the 471st power possible outcomes, so many that writing down even 20% of them would take more than every atom in the universe. Bird described this as a doubly exponential problem, one that grows far faster than the environment itself, which is why no team can reason through it by hand or brute force it with compute.
Bird contrasted Stag’s approach with the ways teams cope today. Some tools limit how many steps ahead they look and brute force what remains, which is expensive. Observability, which grew out of Google’s site reliability engineering (SRE) practices, accepts failure and relies on fast detection and rollback, but some changes, such as a dropped database column, cost far more to undo than to make. Experienced engineers fill the gap with tribal knowledge and idealized mental models, which Bird called a form of sampling that misses corner cases. Stag says its breakthrough runs these simulations on a single computer in under a minute, treats rare and common cases equally, and reaches a probabilistic guarantee of nine nines that it has covered every meaningful outcome using no more than 512 simulations. Bird clarified for delegates that the nine nines refers to coverage confidence, not availability, and said individual simulations are about 98% accurate, with false-positive and false-negative rates of 0.01% each.
Across a few thousand changes at its first customers, Stag found that about 60% were unsafe, with the problems split roughly evenly between infrastructure and code. In one case, Stag detected an engineer quietly restarting an environment overnight that the team believed was shut down. Customers now check Stag’s change history first during an incident, because it records every change and its side effects, including systems that were affected but never reported a problem. Engineers can explain the intent behind a change in Terraform comments or in the Stag interface, and they can add their own context, such as internal policies or peak periods like Black Friday, to shape the simulations. Bird stressed that AI in Stag explains findings to engineers but does not run the simulations, and that Stag works forward from known changes rather than inferring causes backward from an outage the way root cause analysis tools do. He closed by noting that teams using simulation while coding get changes right the first time and spend less time refactoring after deployment.
Personnel: Rick Richardson, Rob Bird
Thank you for being part of the Tech Field Day community! Our mailing list is a great way to stay up to date on our events and technical content, and we appreciate your signup.
We promise that we’ll never spam you, send ads, or sell your information. This list will only be used to communicate with our community about our events and content. And we’ll limit it to no more than one message per week.
Although we only need your email address, it would be nice if you provided a little more information to help us get to know you better!