3 ms·

Single Point of Failure Is PDX still a single-point-of-failure for Cloudflare services? It was 5-months ago [0], and if I understand the post - it sounds like

Single Point of Failure Is PDX still a single-point-of-failure for Cloudflare services? It was 5-months ago [0], and if I understand the post - it sounds like it still is. If anyone knows, I'd be curious to hear. [0] https://news.ycombinator.com/item?id=38113503 https://news.ycombinator.com/item?id=38113503

internetter 3y ago
From what I understand of what I read, it was a single point of failure for one product instead of 15
- CodeWriter23 3y ago
  ...and they were already on fixing the 1.
vlovich123 3y ago
It’s complicated and it’s not supposed to be. PDX is where the config plane for the edge network and services is stored. Some of this information is transported to the edge via QuickSilver [1] (e.g. auth tokens). This information is replicated to Europe and fail over is possible. The challenge with the previous outage was a combination of things (as it always is) whereby certain services had a dependency that relied on PDX as a single point of failure (if I recall correctly). That underpinned enough services that a good chunk of Cloudflare’s config plane went down. Additionally, the cutover to Europe didn’t go smoothly because it was instantaneous instead of gradual which resulted in traffic amplification as retries for previous requests & current requests were shuttled into the online data center resulting in a thundering herd. What this blog post talks about it is how this time nothing went down (or at least cut over within minutes due to presumably automated systems noticing & doing it) with the exception of analytics data which does have a single dependency and that’s determined to be “ok” (i.e. it’s an acceptable failure mode for that product & that product only). There are additional failure complications required because PDX (& Europe) is composed of several independent data centers which aren’t supposed to all fail simultaneously. What’s pretty clear from the implied “we’re not happy either” is that Cloudflare isn’t pleased with their vendor’s separately located data centers still having correlated failures. [1] https://blog.cloudflare.com/introducing-quicksilver-configuration-distribution-at-internet-scale https://blog.cloudflare.com/introducing-quicksilver-configur...
- alberth 3y ago
  > Cloudflare isn’t pleased with their vendor’s separately located data centers still having correlated failures. I hear you, but … … what’s now core fabric of the internet (cloudflare) shouldn’t be suseptable to a single vendors data center going down to - knock out their service. It create a situation where an attacker now knows all they need to do is target that single data center (PDX), cloudflare will go down, and now they can attack any website previously protected by cloudflare. I’m not trying to be a hater. I’m just trying to call out, it’s a bit unfair to solely point the figure at this data center provider when architecturally this issue shouldn’t exist in the first place.
  - MallocVoidstar 3y ago
    >cloudflare will go down, and now they can attack any website previously protected by cloudflare. Neither this outage nor the previous broke Cloudflare's protection.
  - vlovich123 3y ago
    Just to be clear - only the config plane was updated (e.g. making changes to how products worked at the edge, creating new tokens for some services etc). The edge data plane itself is immune to this kind of failure (i.e. no websites protected by cloudflare went down as a result of this).
- ianburrell 3y ago
  BTW, Portland is a bad place to have single point-of-failure data centers. Portland is vulnerable to Cascadia subduction zone quake. The probability is 37% over next 50 years. Which is low each year but risk needs to be considered. It will knock out all local centers and power may not be restored for months.

internetter 3y ago

From what I understand of what I read, it was a single point of failure for one product instead of 15

CodeWriter23 3y ago
...and they were already on fixing the 1.

vlovich123 3y ago

It’s complicated and it’s not supposed to be. PDX is where the config plane for the edge network and services is stored. Some of this information is transported to the edge via QuickSilver [1] (e.g. auth tokens). This information is replicated to Europe and fail over is possible. The challenge with the previous outage was a combination of things (as it always is) whereby certain services had a dependency that relied on PDX as a single point of failure (if I recall correctly). That underpinned enough services that a good chunk of Cloudflare’s config plane went down. Additionally, the cutover to Europe didn’t go smoothly because it was instantaneous instead of gradual which resulted in traffic amplification as retries for previous requests & current requests were shuttled into the online data center resulting in a thundering herd. What this blog post talks about it is how this time nothing went down (or at least cut over within minutes due to presumably automated systems noticing & doing it) with the exception of analytics data which does have a single dependency and that’s determined to be “ok” (i.e. it’s an acceptable failure mode for that product & that product only). There are additional failure complications required because PDX (& Europe) is composed of several independent data centers which aren’t supposed to all fail simultaneously. What’s pretty clear from the implied “we’re not happy either” is that Cloudflare isn’t pleased with their vendor’s separately located data centers still having correlated failures. [1] https://blog.cloudflare.com/introducing-quicksilver-configuration-distribution-at-internet-scale https://blog.cloudflare.com/introducing-quicksilver-configur...

alberth 3y ago
> Cloudflare isn’t pleased with their vendor’s separately located data centers still having correlated failures. I hear you, but … … what’s now core fabric of the internet (cloudflare) shouldn’t be suseptable to a single vendors data center going down to - knock out their service. It create a situation where an attacker now knows all they need to do is target that single data center (PDX), cloudflare will go down, and now they can attack any website previously protected by cloudflare. I’m not trying to be a hater. I’m just trying to call out, it’s a bit unfair to solely point the figure at this data center provider when architecturally this issue shouldn’t exist in the first place.
- MallocVoidstar 3y ago
  >cloudflare will go down, and now they can attack any website previously protected by cloudflare. Neither this outage nor the previous broke Cloudflare's protection.
- vlovich123 3y ago
  Just to be clear - only the config plane was updated (e.g. making changes to how products worked at the edge, creating new tokens for some services etc). The edge data plane itself is immune to this kind of failure (i.e. no websites protected by cloudflare went down as a result of this).
ianburrell 3y ago
BTW, Portland is a bad place to have single point-of-failure data centers. Portland is vulnerable to Cascadia subduction zone quake. The probability is 37% over next 50 years. Which is low each year but risk needs to be considered. It will knock out all local centers and power may not be restored for months.