4 ms·

It did happen, and cloudflare should learn from it, but not just the technical reasons. Instead of focusing on the technical reasons why, they should answer ho

by kjuulh 11mo ago

It did happen, and cloudflare should learn from it, but not just the technical reasons. Instead of focusing on the technical reasons why, they should answer how such a change bubbled out to cause such a massive impact instead. Why: Proxy fails requests Why: Handlers crashed because of OOM Why: Clickhouse returns too much data Why: A change was introduced causing double the amount of data Why: A central change was rolled out immediately to all cluster (single point of failure) Why: There are exemptions or standard operating procedure (gate) for releasing changes to the hot path for cloudflares network infra. While the Clickhouse change is important, I personally think it is crucial that Cloudflare tackles the processes, and possibly gates / controls rollout for hot path system, no matter what kind of change they are when they're at that scale it should be possible. But that is probably enough co-driving. It to me seems like a process issue more than a technical one.

lysace 11mo ago
Very quick rollout is crucial for this kind of service. On top of what you wrote, institutionalizing rollback by default if something catastrophically breaks should be the norm. Been there in those calls, begging to people in charge who perhaps shouldn't have been, "eh, maybe we should attempt a rollback to the last known good state? cause, it, you know.... worked". But investigating further before making any change always seems to be the preferred action to these people. Can't be faulted for being cautious and doing things properly, right? I kid you not - this is their instinct. If I recall correctly it took CF 2 hours to roll back the broken changes. So if I were in charge of Cloudflare (4-5k employees) I'd both look at the processes and the people in charge.
- vablings 11mo ago
  It does seem insane to me that there isnt a process to catch the panic, unwind back to a reasonable place in the call stack, load the last known good configuration and continue execution as normal. You would go from having a global 2 hour outage to a warning on a dashboard that can be investigated in a timely manner rather than blowing up half the internet

by kjuulh 11mo ago

lysace 11mo ago

Very quick rollout is crucial for this kind of service. On top of what you wrote, institutionalizing rollback by default if something catastrophically breaks should be the norm. Been there in those calls, begging to people in charge who perhaps shouldn't have been, "eh, maybe we should attempt a rollback to the last known good state? cause, it, you know.... worked". But investigating further before making any change always seems to be the preferred action to these people. Can't be faulted for being cautious and doing things properly, right? I kid you not - this is their instinct. If I recall correctly it took CF 2 hours to roll back the broken changes. So if I were in charge of Cloudflare (4-5k employees) I'd both look at the processes and the people in charge.

vablings 11mo ago
It does seem insane to me that there isnt a process to catch the panic, unwind back to a reasonable place in the call stack, load the last known good configuration and continue execution as normal. You would go from having a global 2 hour outage to a warning on a dashboard that can be investigated in a timely manner rather than blowing up half the internet