3 ms·
In my 30 years of reliability engineering, I've come to learn that this is a distinction without a difference. People think of configuration updates (or state
by otterley 1y ago
In my 30 years of reliability engineering, I've come to learn that this is a distinction without a difference.
People think of configuration updates (or state updates, call them what you will) as inherently safer than code updates, but history (and today!) demonstrates that they are not. Yet even experienced engineers will allow changes like these into production unattended -- even ones who wouldn't dare let a single line of code go live without being subject to the full CI/CD process.
- HumanOstrich 1y agoThey narrowed down the actual problem to some Rust code in the Bot Management system that enforced a hard limit on the number of configuration items by returning an error, but the caller was just blindly unwrapping it.
- otterley 1y agoA dormant bug in the code is usually a condition precedent to incidents like these. Later, when a bad input is given, the bug then surfaces. The bug could have laid dormant for years or decades, if it ever surfaced at all. The point here remains: consider every change to involve risk, and architect defensively.
- tptacek 1y agoThey made the classic distributed systems mistake and actually did something. Never leap to thing-doing!
- otterley 1y agoIf they're going to yeet configs into production, they ought to at least have plenty of mitigation mechanisms, including canary deployments and fault isolation boundaries. This was my primary point at the root of this thread. And I hope fly.io has these mechanisms as well :-)
- tptacek 1y agoWe've written at long, tedious length about how hard this problem is.
- otterley 1y agoHave a link?
- tptacek 1y agoMost recently, a few weeks ago (but you'll find more just a page or two into the blog): https://fly.io/blog/corrosion/ https://fly.io/blog/corrosion/
- otterley 1y agoIt's great that you're working on regionalization. Yes, it is hard, but 100x harder if you don't start with cellular design in mind. And as I said in the root of the thread, this is a sign that CloudFlare needs to invest in it just like you have been.
- tptacek 1y agoI recoil from that last statement not because I have a rooting interest in Cloudflare but because the last several years of working at Fly.io have drilled Richard Cook's "How Complex Systems Fail"† deep into my brain, and what you said runs aground of Cook #18: Failure free operations require experience with failure. If the exact same thing happens again at Cloudflare, they'll be fair game. But right now I feel people on this thread are doing exactly, precisely, surgically and specifically the thing Richard Cook and the Cook-ites try to get people not to do, which is to see complex system failures as predictable faults with root causes, rather than as part of the process of creating resilient systems. † https://how.complexsystems.fail/ https://how.complexsystems.fail/
- otterley 1y ago
- Ekaros 1y agoSounds like lack of good testing. Too many items in any input should be a boundary case you will get to eventually.
- tptacek 1y agoReframe this problem: instead of bot rules being propagated, it's the enrollment of a new customer or a service at an existing customer --- something that must happen at Cloudflare several times a second. Does it still make sense to you to think about that in terms of "pushing new configuration to prod"?