2 ms·

> We make software deployments constantly across the network and have automated systems to run test suites and a procedure for deploying progressively to preven

> We make software deployments constantly across the network and have automated systems to run test suites and a procedure for deploying progressively to prevent incidents. Good. > Unfortunately, these WAF rules were deployed globally in one go and caused today’s outage. Wow. This seems like a very immature operational stance. Any deployment of any kind should be subject to minimum deployment safety, that they claim they have. > At 1402 UTC we understood what was happening and decided to issue a ‘global kill’ on the WAF Managed Rulesets, which instantly dropped CPU back to normal and restored traffic. That occurred at 1409 UTC. Many large companies would have had automatic roll-back of this kind of change in less time than it took CloudFlare to (apparently) have humans decide to roll-back, and possibly before a single (usually not global) deployment had actually completed on all hosts/instances. However, what is more concerning is that it seems you shouldn't rely on CloudFlare's "WAF Managed Rulesets" at all, since they seem to be willing to turn it off instead of correctly rolling back a bad deployment, which they only did > 43 minutes later: > We then went on to review the offending pull request, roll back the specific rules, test the change to ensure that we were 100% certain that we had the correct fix, and re-enabled the WAF Managed Rulesets at 1452 UTC. How were they not able to trivially roll back to the previous deployment?

londons_explore 7y ago
So many employees deploying so many changes at a time it wasn't clear which one was the cause...?
- pornel 7y ago
  Deployments are scheduled and managed by the SRE team.
- mschuster91 7y ago
  Which is why the entire (mostly in Agile environments) model of "deploy to prod as soon as you can" is absolute nuts. If you're dev at a hipster app maybe a dozen people use to holler "yo" at each other, by all means go for it. If you're operating one of the biggest and most important chonks of Internet infra... maaaaaybe stick to established practices such as stage testing, release schedules and incremental rollouts?
  - flurdy 7y ago
    Why not both. I don't want to return to the old slothful release schedules of the 2000s, where features and bug fixes was mostly stagnant. You can have staging, scheduled QA signed off releases, that happen every day. I have worked on some fairly large significant services and we still released several times a day, just that you did not trigger the final prod release yourself but the QAs pressed the button instead. Though usually just once a day per microservice. I have also worked with several clients lately without QA where devs could themselves push to prod many times a day. I am not sure these systems were that much less stable, though they were all mostly greenfield and not critical public government systems. They were off course a lot smaller changes, and quick to undo. Which is the core element of "release straight to prod" ethos. I am sure Cloudflare have a significant QA process whilst using todays fast moving release schedules. What is always a grey-zone is configuration changes. Even if properly versioned and on a release schedule train with several staging environments, configuration is often very environment sensitive. So maybe they could not test it properly in any staging environments but had to hope prod worked... However Cloudflare will hopefully implement some way to make sure this particular configuration and subsequent future changes are not as bottle-necked that instead can be be gradually released to a subset and region-by-region instead of a big bang to all. Though canary/blue-green/etc releases of core routing configurations is hard.

londons_explore 7y ago

So many employees deploying so many changes at a time it wasn't clear which one was the cause...?

pornel 7y ago
Deployments are scheduled and managed by the SRE team.
mschuster91 7y ago
Which is why the entire (mostly in Agile environments) model of "deploy to prod as soon as you can" is absolute nuts. If you're dev at a hipster app maybe a dozen people use to holler "yo" at each other, by all means go for it. If you're operating one of the biggest and most important chonks of Internet infra... maaaaaybe stick to established practices such as stage testing, release schedules and incremental rollouts?
- flurdy 7y ago
  Why not both. I don't want to return to the old slothful release schedules of the 2000s, where features and bug fixes was mostly stagnant. You can have staging, scheduled QA signed off releases, that happen every day. I have worked on some fairly large significant services and we still released several times a day, just that you did not trigger the final prod release yourself but the QAs pressed the button instead. Though usually just once a day per microservice. I have also worked with several clients lately without QA where devs could themselves push to prod many times a day. I am not sure these systems were that much less stable, though they were all mostly greenfield and not critical public government systems. They were off course a lot smaller changes, and quick to undo. Which is the core element of "release straight to prod" ethos. I am sure Cloudflare have a significant QA process whilst using todays fast moving release schedules. What is always a grey-zone is configuration changes. Even if properly versioned and on a release schedule train with several staging environments, configuration is often very environment sensitive. So maybe they could not test it properly in any staging environments but had to hope prod worked... However Cloudflare will hopefully implement some way to make sure this particular configuration and subsequent future changes are not as bottle-necked that instead can be be gradually released to a subset and region-by-region instead of a big bang to all. Though canary/blue-green/etc releases of core routing configurations is hard.