Deployment shouldn't normally be a huge cause of serious outages because the problems should be caught early, when they are affecting only a small fraction of t

Deployment shouldn't normally be a huge cause of serious outages because the problems should be caught early, when they are affecting only a small fraction of traffic. If you deploy to one of ten thousand replicas, and even if the new build is completely broken, serving errors to 0.0001 of your queries for a few minutes isn't going to eat much of your failure budget. Even better if you can deploy to testing, internal, or other non-public-facing replicas that don't count against your failure budget at all. There are very large services at Google that deploy builds daily. We have a lot of monitoring, and many of our services are rigged to automatically detect and roll back bogus pushes. The largest source of outages in these large systems is not software deployment, but configuration changes especially in systems that aren't rigged for regular automatic push. The second largest source is somewhat ironically incorrect response to small outages.