3 ms·

I’m really curious what their rollout procedure is, because it seems like many of their past outages should have been uncovered if they released these configura

I’m really curious what their rollout procedure is, because it seems like many of their past outages should have been uncovered if they released these configuration changes to 1% of global traffic first.

Traubenfuchs 11mo ago
"Please don‘t block the rollout pipleline with a simple react security patch update."
lima 11mo ago
They don't appear to have a rollout procedure for some of their globally replicated application state. They had a number of major outages over the past years which all had the same root cause of "a global config change exposed a bug in our code and everything blew up". I guess it's an organizational consequence of mitigating attacks in real time, where rollout delays can be risky as well. But if you're going to do that, it would appear that the code has to be written much more defensively than what they're doing it right now.
- tehlike 11mo ago
  You can selectively bypass many roll out procedures in a properly designed system.
  - lima 11mo ago
    If there is a proper rollout procedure that would've caught this, and they bypass it for routine WAF configuration changes, they might as well not have one.
- ethbr1 11mo ago
  For hypothetical conflicting changes (read worst case: unupgraded nodes/services can't interop with upgraded nodes/services), what's best practice for a partial rollout? Blue/green and temporarily ossify capacity? Regional?
  - cryptonym 11mo ago
    - Push a version with the new logic but not yet enabled, still using legacy logic, able to implement both - Push a version that enables new logic for 1% of traffic - Continue rollout until 100%
    - nrhrjrjrjtntbt 11mo ago
      Can also do canary rollout before that. Canary means rollout to endpoints only used by CF to test. Monitor metrics and automated test results.
      - cryptonym 11mo ago
        That's ok but doesn't solve issues you notice only on actual prod traffic. While it can be a nice addition to catch issues earlier with minimal user impact, best practice on large scale systems still requires a staged/progressive prod rollout.
        nrhrjrjrjtntbt 11mo ago
        Yep. This is definitely an "as well as" Unit test, Integration Test, Staging Test, Staging Rollout, Production Test, Canary, Progressive Rollout Can all be automated can smash through all that quickly with no human intervention.
- JB_Dev 11mo ago
  Yea agree.. This is the same discussion point that came up last time they had an incident. I really don’t buy this requirement to always deploy state changes 100% globally immediately. Why can’t they just roll out to 1%, scaling to 100% over 5 minutes (configurable), with automated health checks and pauses? That will go along way towards reducing the impact of these regressions. Then if they really think something is so critical that it goes everywhere immediately, then sure set the rollout to start at 100%. Point is, design the rollout system to give you that flexibility. Routine/non-critical state changes should go through slower ramping rollouts.
  - franktankbank 11mo ago
    Can't get hacked when you are down.
- nrhrjrjrjtntbt 11mo ago
  Not sure I buy it. Do 1% for 10 minutes. I mean it must have taken over half a day to code and test a patch. Why not wait another 10 minutes.
stogot 11mo ago
The update they describe should never bring down all services. I agree with other posters that they must lack a rollout strategy yet they sent spam emails mocking the reliability of other clouds
- deleted 11mo ago
  [deleted]
- brandensilva 11mo ago
  The irony is they support rolling out incrementally with some of their products for deployment. They need that same mindset for themselves in config/updates/infra changes but probably easier said than done.
gpi 11mo ago
I believe they use Argo according to a previous post mortem. https://blog.cloudflare.com/deep-dive-into-cloudflares-sept-12-dashboard-and-api-outage/ https://blog.cloudflare.com/deep-dive-into-cloudflares-sept-...

Traubenfuchs 11mo ago

"Please don‘t block the rollout pipleline with a simple react security patch update."

lima 11mo ago

They don't appear to have a rollout procedure for some of their globally replicated application state. They had a number of major outages over the past years which all had the same root cause of "a global config change exposed a bug in our code and everything blew up". I guess it's an organizational consequence of mitigating attacks in real time, where rollout delays can be risky as well. But if you're going to do that, it would appear that the code has to be written much more defensively than what they're doing it right now.

tehlike 11mo ago
You can selectively bypass many roll out procedures in a properly designed system.
- lima 11mo ago
  If there is a proper rollout procedure that would've caught this, and they bypass it for routine WAF configuration changes, they might as well not have one.
ethbr1 11mo ago
For hypothetical conflicting changes (read worst case: unupgraded nodes/services can't interop with upgraded nodes/services), what's best practice for a partial rollout? Blue/green and temporarily ossify capacity? Regional?
- cryptonym 11mo ago
  - Push a version with the new logic but not yet enabled, still using legacy logic, able to implement both - Push a version that enables new logic for 1% of traffic - Continue rollout until 100%
  - nrhrjrjrjtntbt 11mo ago
    Can also do canary rollout before that. Canary means rollout to endpoints only used by CF to test. Monitor metrics and automated test results.
    - cryptonym 11mo ago
      That's ok but doesn't solve issues you notice only on actual prod traffic. While it can be a nice addition to catch issues earlier with minimal user impact, best practice on large scale systems still requires a staged/progressive prod rollout.
      - nrhrjrjrjtntbt 11mo ago
        Yep. This is definitely an "as well as" Unit test, Integration Test, Staging Test, Staging Rollout, Production Test, Canary, Progressive Rollout Can all be automated can smash through all that quickly with no human intervention.
JB_Dev 11mo ago
Yea agree.. This is the same discussion point that came up last time they had an incident. I really don’t buy this requirement to always deploy state changes 100% globally immediately. Why can’t they just roll out to 1%, scaling to 100% over 5 minutes (configurable), with automated health checks and pauses? That will go along way towards reducing the impact of these regressions. Then if they really think something is so critical that it goes everywhere immediately, then sure set the rollout to start at 100%. Point is, design the rollout system to give you that flexibility. Routine/non-critical state changes should go through slower ramping rollouts.
- franktankbank 11mo ago
  Can't get hacked when you are down.
nrhrjrjrjtntbt 11mo ago
Not sure I buy it. Do 1% for 10 minutes. I mean it must have taken over half a day to code and test a patch. Why not wait another 10 minutes.

stogot 11mo ago

The update they describe should never bring down all services. I agree with other posters that they must lack a rollout strategy yet they sent spam emails mocking the reliability of other clouds

deleted 11mo ago
[deleted]
brandensilva 11mo ago
The irony is they support rolling out incrementally with some of their products for deployment. They need that same mindset for themselves in config/updates/infra changes but probably easier said than done.

gpi 11mo ago

I believe they use Argo according to a previous post mortem. https://blog.cloudflare.com/deep-dive-into-cloudflares-sept-12-dashboard-and-api-outage/ https://blog.cloudflare.com/deep-dive-into-cloudflares-sept-...