4 ms·

Contrary to others here, I find the postmortem a bit lacking. The TLDR is that CF runs in multiple data centers, one went down, and the services that depend on

by TekMol 3y ago

Contrary to others here, I find the postmortem a bit lacking. The TLDR is that CF runs in multiple data centers, one went down, and the services that depend on it went down with it. The interesting question would be why those services did depend on a single data center. They are pretty vague about it Cloudflare allows multiple teams to innovate quickly. As such, products often take different paths toward their initial alpha. If I was the CEO, I would look into the specific decisions of the engineers and why they decided to make services depend on just one data center. That would make an interesting blog post to me. Designing a highly available system and building a company fast leads to interesting tradeoffs. The details would be interesting.

pests 3y ago
Isn't this sentence a bit further down more clear? > This is a system design that we began implementing four years ago. While most of our critical control plane systems had been migrated to the high availability cluster, some services, especially for some newer products, had not yet been added to the high availability cluster. and > It [PDX-04] is also the default location for services that have not yet been onboarded onto our high availability cluster.
nanankcornering 3y ago
I experienced it myself within the last 24 hours. New D1 & Hyperdrive deployment was not working. It would spew out internal errors & timeouts. Both are non-GA products, and the point is that non-GA are not part of the HA cluster (yet)
kccqzy 3y ago
> why they decided to make services depend on just one data center In my experience, no engineers really decided to make services depend on just one data center. It happened because the dependency was overlooked. Or it happened because the dependency was thought to be a "soft dependency" with graceful degradation in case of unavailability but the graceful degradation path had a bug. Or it happened because the engineers thought it had a dependency on one of multiple data centers, but then the failover process had a bug. Reminds me of that time when a single data center in Paris for GCP brought down the entire Google Cloud Console albeit briefly. Really the same thing.
- throwaway6920 3y ago
  > In my experience, no engineers really decided to make services depend on just one data center. Partially true in this case; I can't speak to modern CF (or won't, moreso) but a large amount of internal services were built around SQL db's, and weren't built with any sense of eventual consistency. Usage of read replicas was basically unheard of. Knowing that, and that this was normal, it's a cultural issue rather than an "oops" issue. Flipping the whole DC data sources is a sign of what I'm describing; FAANG would instead be running services in multiple DC's rather than relying on primary/secondary architecture.
  - disgruntledphd2 3y ago
    Dunno about that, I've read similar internal postmortems at the FAANG I worked at.
    - daxfohl 3y ago
      Everywhere I've worked requires a DR drill per service, but I've never seen anything where the whole company shuts down a DC at once across all services. But probably we should. It's an immensely larger coordination problem, but frankly, it's probably the more common failure mode.
      - disgruntledphd2 3y ago
        The FAANG I worked at did this back in 2016-18, so that what happened to CloudFlare didn't happen to them.
nijave 3y ago
>I would look into the specific decisions of the engineers and why they decided to make services depend on just one data center And the product team defining requirements And IT/governance/architecture teams for not properly cataloging dependencies And the sales and marketing team not clearly articulating what they're selling (a beta/early access product that's not HA)

by TekMol 3y ago

pests 3y ago

Isn't this sentence a bit further down more clear? > This is a system design that we began implementing four years ago. While most of our critical control plane systems had been migrated to the high availability cluster, some services, especially for some newer products, had not yet been added to the high availability cluster. and > It [PDX-04] is also the default location for services that have not yet been onboarded onto our high availability cluster.

nanankcornering 3y ago

I experienced it myself within the last 24 hours. New D1 & Hyperdrive deployment was not working. It would spew out internal errors & timeouts. Both are non-GA products, and the point is that non-GA are not part of the HA cluster (yet)

kccqzy 3y ago

> why they decided to make services depend on just one data center In my experience, no engineers really decided to make services depend on just one data center. It happened because the dependency was overlooked. Or it happened because the dependency was thought to be a "soft dependency" with graceful degradation in case of unavailability but the graceful degradation path had a bug. Or it happened because the engineers thought it had a dependency on one of multiple data centers, but then the failover process had a bug. Reminds me of that time when a single data center in Paris for GCP brought down the entire Google Cloud Console albeit briefly. Really the same thing.

throwaway6920 3y ago
> In my experience, no engineers really decided to make services depend on just one data center. Partially true in this case; I can't speak to modern CF (or won't, moreso) but a large amount of internal services were built around SQL db's, and weren't built with any sense of eventual consistency. Usage of read replicas was basically unheard of. Knowing that, and that this was normal, it's a cultural issue rather than an "oops" issue. Flipping the whole DC data sources is a sign of what I'm describing; FAANG would instead be running services in multiple DC's rather than relying on primary/secondary architecture.
- disgruntledphd2 3y ago
  Dunno about that, I've read similar internal postmortems at the FAANG I worked at.
  - daxfohl 3y ago
    Everywhere I've worked requires a DR drill per service, but I've never seen anything where the whole company shuts down a DC at once across all services. But probably we should. It's an immensely larger coordination problem, but frankly, it's probably the more common failure mode.
    - disgruntledphd2 3y ago
      The FAANG I worked at did this back in 2016-18, so that what happened to CloudFlare didn't happen to them.

nijave 3y ago

>I would look into the specific decisions of the engineers and why they decided to make services depend on just one data center And the product team defining requirements And IT/governance/architecture teams for not properly cataloging dependencies And the sales and marketing team not clearly articulating what they're selling (a beta/early access product that's not HA)