I'm sure they use Canary Deployments, Gradual Rollouts and what have you to update their services. I suppose this is a hard problem to solve on a configuration

I'm sure they use Canary Deployments, Gradual Rollouts and what have you to update their services. I suppose this is a hard problem to solve on a configuration change level though. Imagine the configuration change that triggered the bug was something like "hey load balancers, stop sending traffic to the cluster with that new version of service X which seems to cause elevated error rates." You don't really want that kind of change to take too long to propagate.