This is a really great question since the vast majority of the work was in ensuring correctness and reliability: everything from testing discipline to fault inj

This is a really great question since the vast majority of the work was in ensuring correctness and reliability: everything from testing discipline to fault injection to auditing. This also included hardware testing, like pulling out circuit breakers to test our power distribution, or overheating a rack to test graceful shutdown. I'll give a slightly lazy answer here however and point you to a talk I gave about building durable systems, which covers a lot of this material: https://www.oreilly.com/events/velocity/devops-web-performance-ny-2015/sessions/46120/durability-theater-or-real-world-safety-how-to-engineer-against-data-loss https://www.oreilly.com/events/velocity/devops-web-performan...