4 ms·

Heh. They needed Workers KV running to get Workers KV running.

ryanjshaw 3y ago
They say they needed to use a manual/break glass process to get things going again. Kinda scary to think Cloudflare could brick itself.
- swatcoder 3y ago
  > Kinda scary to think Cloudflare could brick itself. Never forget that essentially all interesting systems may brick themselves and will at least catastrophically fail every now and then. For all the sales pitches about "the cloud" providing resiliency, it's fundamentally centralizes an ungodly share of the internet onto a handful of failure points. What had been a frothy background noise of independent hosting and data center failures has become a global blackout risk, where worldwide operations break down and failures cascade through interdependent systems. On a business level, it's hard to care about that, as you're not going to be held to blame if everything else goes into crisis at the same time as you, but on a societal level, it's kinda f---'d. It's like moving all the agriculture onto the flood plain. You get maximum abundance most of the time, but when it fails in those inevitable floods, you at least better have kept some stockpiles stored uphill somewhere.
- starttoaster 3y ago
  They don't say what the manual/break glass process was exactly. But I likened it to something like a `helm rollback` for a kubernetes helm release. Manual/break-glass doesn't have to mean a team of engineers are logged into VM instances `vi` editing config files back to a previous state and `scp`-ing older binaries over, it just probably means there wasn't an automated build/deploy tool process to perform it. > Kinda scary to think Cloudflare could brick itself. I mean.. They are providers of software that iterate on their product suite in a SaaS environment. I think this sentiment holds for pretty much any SaaS company. And I guess if everything is scary then nothing is really all that scary. Do you disagree?
  - ryanjshaw 3y ago
    I meant "brick" as distinct from "break", in the sense that their apparent circular dependencies could result in an unrecoverable failure scenario. Purely speculation of course, it's probably not a real risk. That being said, it's a scary thought because they are different from other SaaS in that the run a significant portion of the internet.
    - pixl97 3y ago
      I mean near brick situations happen with DNS/VM/Auth situations quite often. It's not hard to imagine where some dumbass creates a system that also contains its own decryption keys in some system and it "will be fine because all auth systems will never be down".
    - starttoaster 3y ago
      There were no apparent circular dependencies. Services that Cloudflare offer depend on their KV store service. A KV store (or key-value store) is a type of database. If you have an application that depends on a database, that's a one-way dependency, not a circular one.
- wbl 3y ago
  You know how cell sites have gas generators and the gas line uses cell to communicate status? I've always wondered about that one. Plus would a diesel seller be able to take cash if a data center needed it to start up again. There's plenty more where that thoughts from
  - thedaly 3y ago
    I don't know about cell towers, but all AT&T central offices have generators with massive fuel tanks that can run even if the gas supply is cut off. I would be surprised if the backup for cell towers didn't have similar systems.
    - Arrath 3y ago
      My partner's father works for AT&T and he has related a story where at least one cell tower went offline because the office was ignoring the NO LINE POWER/BACKUP GEN FUEL LOW warning duo[0] and assumed all was good when the warnings went silent when, in fact, it was now dark. [0]A very rural installation that often experienced power cuts, so they had become numb to alarm 1. Somewhere in the mystical ticket/service system, the schedule to refuel the tower had go awry so it gradually sucked down its fuel over the course of many power interruptions and viola.
remram 3y ago
I worry a lot about those things. I am sending this to my colleagues. It's too easy to set up new sysop tools on the shiny new cluster/filesystem/cloud that makes everything easy. However how do you fix that shiny but complex thing if your tools are down?

They say they needed to use a manual/break glass process to get things going again. Kinda scary to think Cloudflare could brick itself.

swatcoder 3y ago
> Kinda scary to think Cloudflare could brick itself. Never forget that essentially all interesting systems may brick themselves and will at least catastrophically fail every now and then. For all the sales pitches about "the cloud" providing resiliency, it's fundamentally centralizes an ungodly share of the internet onto a handful of failure points. What had been a frothy background noise of independent hosting and data center failures has become a global blackout risk, where worldwide operations break down and failures cascade through interdependent systems. On a business level, it's hard to care about that, as you're not going to be held to blame if everything else goes into crisis at the same time as you, but on a societal level, it's kinda f---'d. It's like moving all the agriculture onto the flood plain. You get maximum abundance most of the time, but when it fails in those inevitable floods, you at least better have kept some stockpiles stored uphill somewhere.
starttoaster 3y ago
They don't say what the manual/break glass process was exactly. But I likened it to something like a `helm rollback` for a kubernetes helm release. Manual/break-glass doesn't have to mean a team of engineers are logged into VM instances `vi` editing config files back to a previous state and `scp`-ing older binaries over, it just probably means there wasn't an automated build/deploy tool process to perform it. > Kinda scary to think Cloudflare could brick itself. I mean.. They are providers of software that iterate on their product suite in a SaaS environment. I think this sentiment holds for pretty much any SaaS company. And I guess if everything is scary then nothing is really all that scary. Do you disagree?
- ryanjshaw 3y ago
  I meant "brick" as distinct from "break", in the sense that their apparent circular dependencies could result in an unrecoverable failure scenario. Purely speculation of course, it's probably not a real risk. That being said, it's a scary thought because they are different from other SaaS in that the run a significant portion of the internet.
  - pixl97 3y ago
    I mean near brick situations happen with DNS/VM/Auth situations quite often. It's not hard to imagine where some dumbass creates a system that also contains its own decryption keys in some system and it "will be fine because all auth systems will never be down".
  - starttoaster 3y ago
    There were no apparent circular dependencies. Services that Cloudflare offer depend on their KV store service. A KV store (or key-value store) is a type of database. If you have an application that depends on a database, that's a one-way dependency, not a circular one.
wbl 3y ago
You know how cell sites have gas generators and the gas line uses cell to communicate status? I've always wondered about that one. Plus would a diesel seller be able to take cash if a data center needed it to start up again. There's plenty more where that thoughts from
- thedaly 3y ago
  I don't know about cell towers, but all AT&T central offices have generators with massive fuel tanks that can run even if the gas supply is cut off. I would be surprised if the backup for cell towers didn't have similar systems.
  - Arrath 3y ago
    My partner's father works for AT&T and he has related a story where at least one cell tower went offline because the office was ignoring the NO LINE POWER/BACKUP GEN FUEL LOW warning duo[0] and assumed all was good when the warnings went silent when, in fact, it was now dark. [0]A very rural installation that often experienced power cuts, so they had become numb to alarm 1. Somewhere in the mystical ticket/service system, the schedule to refuel the tower had go awry so it gradually sucked down its fuel over the course of many power interruptions and viola.

remram 3y ago

I worry a lot about those things. I am sending this to my colleagues. It's too easy to set up new sysop tools on the shiny new cluster/filesystem/cloud that makes everything easy. However how do you fix that shiny but complex thing if your tools are down?