4 ms·
They don't say what the manual/break glass process was exactly. But I likened it to something like a `helm rollback` for a kubernetes helm release. Manual/break
by starttoaster 3y ago
They don't say what the manual/break glass process was exactly. But I likened it to something like a `helm rollback` for a kubernetes helm release. Manual/break-glass doesn't have to mean a team of engineers are logged into VM instances `vi` editing config files back to a previous state and `scp`-ing older binaries over, it just probably means there wasn't an automated build/deploy tool process to perform it.
> Kinda scary to think Cloudflare could brick itself.
I mean.. They are providers of software that iterate on their product suite in a SaaS environment. I think this sentiment holds for pretty much any SaaS company. And I guess if everything is scary then nothing is really all that scary. Do you disagree?
- ryanjshaw 3y agoI meant "brick" as distinct from "break", in the sense that their apparent circular dependencies could result in an unrecoverable failure scenario. Purely speculation of course, it's probably not a real risk. That being said, it's a scary thought because they are different from other SaaS in that the run a significant portion of the internet.
- pixl97 3y agoI mean near brick situations happen with DNS/VM/Auth situations quite often. It's not hard to imagine where some dumbass creates a system that also contains its own decryption keys in some system and it "will be fine because all auth systems will never be down".
- starttoaster 3y agoThere were no apparent circular dependencies. Services that Cloudflare offer depend on their KV store service. A KV store (or key-value store) is a type of database. If you have an application that depends on a database, that's a one-way dependency, not a circular one.