4 ms·

> Any Googlers around to explain why Canaries didn't catch configuration causing this side effect and stopping the deployment before the issue cascaded globally

> Any Googlers around to explain why Canaries didn't catch configuration causing this side effect and stopping the deployment before the issue cascaded globally? I'm not (any longer) a Googler, but according to the postmortem this was a load-related incident. Canary changes only have a small amount of traffic directed at them (by definition) and the problem probably didn't become apparent until after the change was promoted from canary.

j_coder 8y ago
So Google don't test for load? They should ;)
- londons_explore 8y ago
  Most peoples load tests involve taking a system to breaking point, then turning the load off and going to lunch. People need to gradually reduce the load after the breaking point to check the system recovers. Load Tests are also pretty hard to do in distributed systems. If you test the application alone, you probably won't find most of the issues. You'll need to test the application complete with all it's dependencies (databases, load balancers, failover mechanisms, external servers, etc.). You'll also probably want to test it with representative user requests, and all databases filled with representative data. That in turn typically means your loadtest system will need to be as big and expensive as your production system. Have fun explaining to the boss why you need to double the infrastructure costs. If you do it in the cloud, you can just do a loadtest for 10 mins per day and save a bunch of $$$, but you still need tooling to be able to deploy a complete replica of your system, fill it with realistic data and send it realistic requests, all automatically. Using real user data and logs of user interactions and real user requests is best for loadtesting, but comes with it's own risks. You need to make sure the loadtest systems doesn't send out emails to users, or accidentally communicate with the real production systems in any way. It also means you have to secure your loadtest infrastructure as well as your production infrastructure. GDPR data deletion requests need to apply there too, etc.
londons_explore 8y ago
Going further... Googles 'Canary' system is sub-optimal and risky. It picks a subset of the system (say 5%), and deploys the change there. It then observes this 5% of the system, and if it's okay, it rolls out to the rest of the system. Compare that to a continual-canarying system: * Start deploying the new code gradually (say 1 container/pod/task every second). * While the above is in progress, continually collect stats from upgraded and non-upgraded systems. * If the stats differ to any significant level, halt the rollout. The continual-canarying system is much better because the sample size to see a 'significant' change depends on the size of the change and the amount of natural variation in the sample. The size of the change obviously can't be known beforehand. The variation varies widely depending on the metric of interest (cpu, memory use, request latency, etc.). It is therefore impossible in googles canarying system to choose a suitable subset size to detect (with a sufficiently high statistical power) small but important changes in a metric while still stopping the rollout early enough for catastrophic changes in a metric. Continual-canarying effectively solves this issue by gradually increasing the sample size as the rollout goes on. In reality, neither continual-canarying or google-canarying work well for detecting anomalous metrics because the rollout process itself causes applications to restart, which in itself changes their performance characteristics (cold caches, empty queues, first-use delay, jit optimization, etc.). The solution to that is to do two simultaneous continuous-canary rollouts - one of the existing version (effectively just a restart) and one of the new version, and use those two groups as input to the logic to decide if any metrics of interest have statistically significant changes. Please go implement this stuff Google! It won't be hard to do, and will really increase the safety of your rollouts. (Even though I know it wouldn't have helped this time)
- Scaevolus 8y ago
  Those gradual rollouts exist, but sometimes you have changes that aren't so easy to deploy a fraction of a percent at a time.
- joshuamorton 8y ago
  So you'd think this would be the case. I certainly did when I first joined. In fact, I actually proposed something even more extreme than your suggestion (use thompson sampling to control the rate of task restarts and rollbacks). But in practice such a continual canarying process isn't actually any better than a staged canary (like .1% -> 1% -> 10%). Consider the types of issues you run into, there are - Things that are painful, but not destructive (minor performance regressions) - Things that are highly destructive (data gets deleted, major performance regressions, etc.) The first you can handle being deployed to many people, so if you detect at 7.8% of your users instead of 10% doesn't much matter, you can run it at 10% forever without issue. The second you'll detect on a smaller population because the changes are catastrophic. >In reality, neither continual-canarying or google-canarying work well for detecting anomalous metrics because the rollout process itself causes applications to restart, which in itself changes their performance characteristics (cold caches, empty queues, first-use delay, jit optimization, etc.). The problem here is that most of the time, you don't care about startup behavior, but steady-state behavior. Imagine that a new version introduces a memory leak, so the old behavior was to linearly increase to 1GB of memory over 1 hour and then level off, and the new behavior is to increase unbounded until eventually the system OOMs and restarts or whatever. A "google style" canary would roll out to 1% of tasks or something, wait a few hours, and notice the difference and roll back. You might even experience 1% of tasks restating, but it's likely that the system can sustain that. With the continuous canary, you'll release to 1 task per minute or whatever, and only be able to notice any change after you've pushed out to 60 tasks, and the change will likely only be detectable with any confidence once you have a notable difference in 120 or so. At that point, you've released to more than 1% of your tasks (either that, or it takes you 8 days to release a new version). You can fix that by slowing the rate of releases, but now it takes you 8 weeks to release a new version. Plus, even worse, you're now in a much more fail-open environment. With a 1% release, you can sustain in the kind of bad state if all of your qualification tooling fails. If you're continuously canarying though, you have to be much more careful to make sure that your tooling won't continue to push if the tooling itself is broken or getting unusual results. It's a more risky set up.

j_coder 8y ago

So Google don't test for load? They should ;)

londons_explore 8y ago
Most peoples load tests involve taking a system to breaking point, then turning the load off and going to lunch. People need to gradually reduce the load after the breaking point to check the system recovers. Load Tests are also pretty hard to do in distributed systems. If you test the application alone, you probably won't find most of the issues. You'll need to test the application complete with all it's dependencies (databases, load balancers, failover mechanisms, external servers, etc.). You'll also probably want to test it with representative user requests, and all databases filled with representative data. That in turn typically means your loadtest system will need to be as big and expensive as your production system. Have fun explaining to the boss why you need to double the infrastructure costs. If you do it in the cloud, you can just do a loadtest for 10 mins per day and save a bunch of $$$, but you still need tooling to be able to deploy a complete replica of your system, fill it with realistic data and send it realistic requests, all automatically. Using real user data and logs of user interactions and real user requests is best for loadtesting, but comes with it's own risks. You need to make sure the loadtest systems doesn't send out emails to users, or accidentally communicate with the real production systems in any way. It also means you have to secure your loadtest infrastructure as well as your production infrastructure. GDPR data deletion requests need to apply there too, etc.

londons_explore 8y ago

Going further... Googles 'Canary' system is sub-optimal and risky. It picks a subset of the system (say 5%), and deploys the change there. It then observes this 5% of the system, and if it's okay, it rolls out to the rest of the system. Compare that to a continual-canarying system: * Start deploying the new code gradually (say 1 container/pod/task every second). * While the above is in progress, continually collect stats from upgraded and non-upgraded systems. * If the stats differ to any significant level, halt the rollout. The continual-canarying system is much better because the sample size to see a 'significant' change depends on the size of the change and the amount of natural variation in the sample. The size of the change obviously can't be known beforehand. The variation varies widely depending on the metric of interest (cpu, memory use, request latency, etc.). It is therefore impossible in googles canarying system to choose a suitable subset size to detect (with a sufficiently high statistical power) small but important changes in a metric while still stopping the rollout early enough for catastrophic changes in a metric. Continual-canarying effectively solves this issue by gradually increasing the sample size as the rollout goes on. In reality, neither continual-canarying or google-canarying work well for detecting anomalous metrics because the rollout process itself causes applications to restart, which in itself changes their performance characteristics (cold caches, empty queues, first-use delay, jit optimization, etc.). The solution to that is to do two simultaneous continuous-canary rollouts - one of the existing version (effectively just a restart) and one of the new version, and use those two groups as input to the logic to decide if any metrics of interest have statistically significant changes. Please go implement this stuff Google! It won't be hard to do, and will really increase the safety of your rollouts. (Even though I know it wouldn't have helped this time)

Scaevolus 8y ago
Those gradual rollouts exist, but sometimes you have changes that aren't so easy to deploy a fraction of a percent at a time.
joshuamorton 8y ago
So you'd think this would be the case. I certainly did when I first joined. In fact, I actually proposed something even more extreme than your suggestion (use thompson sampling to control the rate of task restarts and rollbacks). But in practice such a continual canarying process isn't actually any better than a staged canary (like .1% -> 1% -> 10%). Consider the types of issues you run into, there are - Things that are painful, but not destructive (minor performance regressions) - Things that are highly destructive (data gets deleted, major performance regressions, etc.) The first you can handle being deployed to many people, so if you detect at 7.8% of your users instead of 10% doesn't much matter, you can run it at 10% forever without issue. The second you'll detect on a smaller population because the changes are catastrophic. >In reality, neither continual-canarying or google-canarying work well for detecting anomalous metrics because the rollout process itself causes applications to restart, which in itself changes their performance characteristics (cold caches, empty queues, first-use delay, jit optimization, etc.). The problem here is that most of the time, you don't care about startup behavior, but steady-state behavior. Imagine that a new version introduces a memory leak, so the old behavior was to linearly increase to 1GB of memory over 1 hour and then level off, and the new behavior is to increase unbounded until eventually the system OOMs and restarts or whatever. A "google style" canary would roll out to 1% of tasks or something, wait a few hours, and notice the difference and roll back. You might even experience 1% of tasks restating, but it's likely that the system can sustain that. With the continuous canary, you'll release to 1 task per minute or whatever, and only be able to notice any change after you've pushed out to 60 tasks, and the change will likely only be detectable with any confidence once you have a notable difference in 120 or so. At that point, you've released to more than 1% of your tasks (either that, or it takes you 8 days to release a new version). You can fix that by slowing the rate of releases, but now it takes you 8 weeks to release a new version. Plus, even worse, you're now in a much more fail-open environment. With a 1% release, you can sustain in the kind of bad state if all of your qualification tooling fails. If you're continuously canarying though, you have to be much more careful to make sure that your tooling won't continue to push if the tooling itself is broken or getting unusual results. It's a more risky set up.