4 ms·
It seems they had this continous rollout for the config service, but the services consuming this were affected even by small percentage of these config provider
by mewpmewp2 11mo ago
It seems they had this continous rollout for the config service, but the services consuming this were affected even by small percentage of these config providers being faulty, since they were auto updating every few minutes their configs. And it seems there is a reason for these updating so fast, presumably having to react to threat actors quickly.
- otterley 11mo agoIt's in everyone's interest to mitigate threats as quickly as possible. But it's of even greater interest that a core global network infrastructure service provider not DOS a significant proportion of the Internet by propagating a bad configuration too quickly. The key here is to balance responsiveness against safety, and I'm not sure they struck the right balance here. I'm just glad that the impact wasn't as long and as severe as it could have been.
- tptacek 11mo agoThis isn't really "configuration" so much as it is "durable state" within the context of this system.
- otterley 11mo agoIn my 30 years of reliability engineering, I've come to learn that this is a distinction without a difference. People think of configuration updates (or state updates, call them what you will) as inherently safer than code updates, but history (and today!) demonstrates that they are not. Yet even experienced engineers will allow changes like these into production unattended -- even ones who wouldn't dare let a single line of code go live without being subject to the full CI/CD process.
- HumanOstrich 11mo agoThey narrowed down the actual problem to some Rust code in the Bot Management system that enforced a hard limit on the number of configuration items by returning an error, but the caller was just blindly unwrapping it.
- otterley 11mo agoA dormant bug in the code is usually a condition precedent to incidents like these. Later, when a bad input is given, the bug then surfaces. The bug could have laid dormant for years or decades, if it ever surfaced at all. The point here remains: consider every change to involve risk, and architect defensively.
- tptacek 11mo agoThey made the classic distributed systems mistake and actually did something. Never leap to thing-doing!
- otterley 11mo agoIf they're going to yeet configs into production, they ought to at least have plenty of mitigation mechanisms, including canary deployments and fault isolation boundaries. This was my primary point at the root of this thread. And I hope fly.io has these mechanisms as well :-)
- tptacek 11mo agoReframe this problem: instead of bot rules being propagated, it's the enrollment of a new customer or a service at an existing customer --- something that must happen at Cloudflare several times a second. Does it still make sense to you to think about that in terms of "pushing new configuration to prod"?