I'm not sure (at least on the rollout side which was what caused the largest impact)... My team does a lot of releases and rollouts. We use automation in the m

I'm not sure (at least on the rollout side which was what caused the largest impact)... My team does a lot of releases and rollouts. We use automation in the majority of cases and automatically pause if alert conditions occur during the rollout. We usually release a dev version (which we can run tests (incl. typical traffic that we expect the service to handle)), canary version and production version. Ideally you want to catch this sort of stuff during the tests that you run against your dev instances. Of course, you have to have the right testing infrastructure, monitoring and alerting in place to avail of that. For some of our services, we use graduated rollouts so we can gradually direct traffic from the old version of the stack to the new one over a period of time. This allows us (where us = our monitoring) to quickly identify if there are issues (e.g. can compare error rates of version x with version x+1) and reset the percentage of traffic going to version x+1 to 0% if there is an issue. From the postmortem, they indicated the need to add additional monitoring to identify an increase in error rate + traffic rate. I'm pretty sure the Production Readiness Reviews called out in the SRE book by Google (see here: https://landing.google.com/sre/book/chapters/evolving-sre-engagement-model.html https://landing.google.com/sre/book/chapters/evolving-sre-en...) would have caught this. The book also has some good insights on how to do better monitoring (hint: don't have humans eyeballing dashboards). Saying all that, I've seen plenty of small bugs make it to that stage of deployment with that level of impact so it's certainly not trivial to get 100% test / monitoring coverage... Reviewing the causes of outages and specifically identifying and resolving action items from them go a long way to ensuring you improve over time.