Almost all outages are due to changes. Bugs in code usually surface pretty quickly, and hardware issues don't pose a problem in such large-scale infrastructure.

Almost all outages are due to changes. Bugs in code usually surface pretty quickly, and hardware issues don't pose a problem in such large-scale infrastructure. Changes causing outages is par for the course. That's why the industry standard is now Infrastructure-as-Code and Immutable Infrastructure: you try to prevent anything from changing in a way that will later break, and manage it in ways that will be easier to fix when it does break. Tools like Terraform are popular today because they allow the planning and staging of changes across complex services. They're still pretty limited, but mapping out dependencies and simulating changes can surface errors before you run into them, thus making it less necessary to perform a rollback. But unexpected problems still happen, which is why you need to test your rollbacks, and intentionally stress random parts of your system to discover unknown bottlenecks. Part of the purpose for stress testing is to have a realistic idea of what kind of capacity you will really have under different conditions. But it's also nearly impossible to accurately stress test production systems without consequences. There are ways for Google to look for change issues, and they probably have lots of safeguards in place, but we don't know what they actually do to test changes. Some of their postmortems have pointed at a lack of stringent change control procedures. Hopefully they will practice what they preach (open/blameless postmortems) and share more details soon.