3 ms·

Wow they REALLY buried this important part didn't they! This took a ton of scrolling: "Unfortunately, we discovered that a subset of services that were suppose

Wow they REALLY buried this important part didn't they! This took a ton of scrolling: "Unfortunately, we discovered that a subset of services that were supposed to be on the high availability cluster had dependencies on services exclusively running in PDX-04." Bingo, there we have it.

dmix 3y ago
What does PDX-04 mean here? Not familiar with how data centers work.
- weird-eye-issue 3y ago
  Read the damn article! It's explained at the top.
  - deleted 3y ago
    [deleted]
- edaemon 3y ago
  PDX is the airport code for Portland, Oregon, USA. It's the fourth Portland data center.
dclowd9901 3y ago
Yep, past the part where they spent a long time blaming the data center and power company.
worksonmine 3y ago
Nah, if only the data center would've stayed up this wouldn't have been a problem. It's clearly on the data center. /s
indigomm 3y ago
In my experience, power is the most common data center failure there is. Often it's the redundant systems that cause the failure.
- weird-eye-issue 3y ago
  And that's completely unrelated to my comment but thanks for the insight
- ta1243 3y ago
  Nobody cares why a data centre died. It's like complaining one of your nodes in a kubernetes cluster has died, or one of your disks in a raid. The problem here, which is 100% Cloudflare, is that their systems were not resilient across geography.
mananaysiempre 3y ago
There’s also the part where the disaster recovery site apparently fell over under the load (which, OK, is a thing that might happen) and they needed to code up limits on the fly (and that is not OK; I don’t have the slightest idea how one might test this, but if you’re building a “disaster” site it seems like you’d need to figure it out): > When services were turned up there, we experienced a thundering herd problem where the API calls that had been failing overwhelmed our services. We implemented rate limits to get the request volume under control. This seems not to be mentioned in the bullet points at the end of the text (which are otherwise reasonable). And now I’m curious—how do you design cold failover when the system is complex enough to be metastable[1] and you can’t afford to test it on live traffic? I can guess which techniques you could use to build it, it’s the design and testing part (knowing the techniques actually work in your situation) that’s the problem. One other thing that seems to have gone completely unmentioned: > Beginning on Thursday, November 2, 2023, at 11:43 UTC Cloudflare's control plane and analytics services experienced an outage. [... W]e made the call at 13:40 UTC to fail over to Cloudflare's disaster recovery sites located in Europe. Why did the decision take so long? I can imagine it can’t be made lightly, but two hours seems like too much hesitation, even if there was an expectation that power would be restored imminently for most of that time. There has to be a (predetermined?) point when you hit the switch regardless of any promises. Was it really set that far? [1] http://charap.co/metastable-failures-in-distributed-systems/ http://charap.co/metastable-failures-in-distributed-systems/

dmix 3y ago

What does PDX-04 mean here? Not familiar with how data centers work.

weird-eye-issue 3y ago
Read the damn article! It's explained at the top.
- deleted 3y ago
  [deleted]
edaemon 3y ago
PDX is the airport code for Portland, Oregon, USA. It's the fourth Portland data center.

dclowd9901 3y ago

Yep, past the part where they spent a long time blaming the data center and power company.

worksonmine 3y ago

Nah, if only the data center would've stayed up this wouldn't have been a problem. It's clearly on the data center. /s

indigomm 3y ago

In my experience, power is the most common data center failure there is. Often it's the redundant systems that cause the failure.

weird-eye-issue 3y ago
And that's completely unrelated to my comment but thanks for the insight
ta1243 3y ago
Nobody cares why a data centre died. It's like complaining one of your nodes in a kubernetes cluster has died, or one of your disks in a raid. The problem here, which is 100% Cloudflare, is that their systems were not resilient across geography.

mananaysiempre 3y ago

There’s also the part where the disaster recovery site apparently fell over under the load (which, OK, is a thing that might happen) and they needed to code up limits on the fly (and that is not OK; I don’t have the slightest idea how one might test this, but if you’re building a “disaster” site it seems like you’d need to figure it out): > When services were turned up there, we experienced a thundering herd problem where the API calls that had been failing overwhelmed our services. We implemented rate limits to get the request volume under control. This seems not to be mentioned in the bullet points at the end of the text (which are otherwise reasonable). And now I’m curious—how do you design cold failover when the system is complex enough to be metastable[1] and you can’t afford to test it on live traffic? I can guess which techniques you could use to build it, it’s the design and testing part (knowing the techniques actually work in your situation) that’s the problem. One other thing that seems to have gone completely unmentioned: > Beginning on Thursday, November 2, 2023, at 11:43 UTC Cloudflare's control plane and analytics services experienced an outage. [... W]e made the call at 13:40 UTC to fail over to Cloudflare's disaster recovery sites located in Europe. Why did the decision take so long? I can imagine it can’t be made lightly, but two hours seems like too much hesitation, even if there was an expectation that power would be restored imminently for most of that time. There has to be a (predetermined?) point when you hit the switch regardless of any promises. Was it really set that far? [1] http://charap.co/metastable-failures-in-distributed-systems/ http://charap.co/metastable-failures-in-distributed-systems/