Sometimes the only way to try out a distributed system is to run it in prod and see what happens. Having the tools to flip behaviour within 1 second globally ca

Sometimes the only way to try out a distributed system is to run it in prod and see what happens. Having the tools to flip behaviour within 1 second globally can be a useful escape hatch. When you get to large enough scales “just roll back” is not always good enough. I deploy systems with tens of thousands of nodes and we specifically have to rate limit how fast we deploy so we don’t cause thundering herds.