3 ms·

Cloudflare's control plane and analytics systems run primarily on servers in three data centers around Hillsboro, Oregon. The three data centers are independent

Cloudflare's control plane and analytics systems run primarily on servers in three data centers around Hillsboro, Oregon. The three data centers are independent of one another, each have multiple utility power feeds, and each have multiple redundant and independent network connections. The facilities were intentionally chosen to be at a distance apart that would minimize the chances that a natural disaster would cause all three to be impacted, while still close enough that they could all run active-active redundant data clusters. If the three data centers are all around Hillsboro, Oregon, an earthquake could probably take out all three simultaneously.

nanankcornering 3y ago
thats why DRC exists right?
pests 3y ago
> Hillsboro, Oregon, an earthquake could probably take out all three simultaneously. Is it west of I5? (yes) Oh yeah, they all gone. Cascadia Subduction Zone - https://pnsn.org/outreach/earthquakesources/csz https://pnsn.org/outreach/earthquakesources/csz
- bell-cot 3y ago
  Wikipedia's entry for Hillsboro: "Elevation 194 ft (60 m)" Between that, and being ~50 miles inland - I'd say there's ~zero threat of Cascadia quakes or tsunamis directly knocking out those DC's. (Yeah, larger-scale infrastructure and social order could still be killers.) OTOH - Mt. St. Helens is about 60 miles NNE of Hillsboro. If that really went boom, and the wind was right...how many cm's of dry volcanic ash can the roofs of those DC's bear? What if rain wets that ash? How about their HVAC systems' filters?
  - pests 3y ago
    I was never worried about the tsunami. Okay, maybe not gone, but I wouldn't say it would be operational. https://www.oregon.gov/oem/Documents/Cascadia_Rising_Exercise_Scenario.pdf https://www.oregon.gov/oem/Documents/Cascadia_Rising_Exercis... 50% of roads and near 75% of bridges damaged on the west coast and the I5 corridor. Refer to PDF page #93 where over 70% of power generation is highly damaged on the I5 corridor and 60% in the coastal areas with 0% undamaged. Highly damaged - "Extensive damage to generation plants, substations, and buildings. Repairs are needed to regain functionality. Restoring power to meet 90% of demand may take months to one year." "In the immediate aftermath of the earthquake, cities within 100 miles of the Pacific coastline may experience partial or complete blackout. Seventy percent of the electric facilities in the I-5 corridor may suffer considerable damage to generation plants, and many distribution circuits and substations may fail, resulting in a loss of over half of the systems load capacity (see Table 22). Most electrical power assets on the coast may suffer damage severe enough as to render the equipment and structures irreparable"
    - bell-cot 3y ago
      Good backups generators at their colo's could handle the lack of utility power for days to weeks. More & better generators could be hauled in and connected.* The two big problems I'd see would be (1) Social Order and (2) Internet Connectivity. DC's are not fortresses, and internet backbone fibers/routers/etc. are distributed & kinda fragile. *After all the large-scale power outages & near-outages of recent decades, Cloudflare has no excuse if they lack really-good backup generators at critical facilities. And with their size, Cloudflare must support enough "critical during major disaster" internet services to actually get such generators.
- ZiiS 3y ago
  And most of thier SREs. Spending 30 hours to recover from the worst natural disaster in recorded history is slightly diffrent then from a ground fault on a single transformer.
- fanf2 3y ago
  There’s also the Portland Hills and Gales Creek fault zones to worry about.
ahoka 3y ago
They would then go into very detailed description how tectonic activity caused the outage.

nanankcornering 3y ago

thats why DRC exists right?

pests 3y ago

> Hillsboro, Oregon, an earthquake could probably take out all three simultaneously. Is it west of I5? (yes) Oh yeah, they all gone. Cascadia Subduction Zone - https://pnsn.org/outreach/earthquakesources/csz https://pnsn.org/outreach/earthquakesources/csz

bell-cot 3y ago
Wikipedia's entry for Hillsboro: "Elevation 194 ft (60 m)" Between that, and being ~50 miles inland - I'd say there's ~zero threat of Cascadia quakes or tsunamis directly knocking out those DC's. (Yeah, larger-scale infrastructure and social order could still be killers.) OTOH - Mt. St. Helens is about 60 miles NNE of Hillsboro. If that really went boom, and the wind was right...how many cm's of dry volcanic ash can the roofs of those DC's bear? What if rain wets that ash? How about their HVAC systems' filters?
- pests 3y ago
  I was never worried about the tsunami. Okay, maybe not gone, but I wouldn't say it would be operational. https://www.oregon.gov/oem/Documents/Cascadia_Rising_Exercise_Scenario.pdf https://www.oregon.gov/oem/Documents/Cascadia_Rising_Exercis... 50% of roads and near 75% of bridges damaged on the west coast and the I5 corridor. Refer to PDF page #93 where over 70% of power generation is highly damaged on the I5 corridor and 60% in the coastal areas with 0% undamaged. Highly damaged - "Extensive damage to generation plants, substations, and buildings. Repairs are needed to regain functionality. Restoring power to meet 90% of demand may take months to one year." "In the immediate aftermath of the earthquake, cities within 100 miles of the Pacific coastline may experience partial or complete blackout. Seventy percent of the electric facilities in the I-5 corridor may suffer considerable damage to generation plants, and many distribution circuits and substations may fail, resulting in a loss of over half of the systems load capacity (see Table 22). Most electrical power assets on the coast may suffer damage severe enough as to render the equipment and structures irreparable"
  - bell-cot 3y ago
    Good backups generators at their colo's could handle the lack of utility power for days to weeks. More & better generators could be hauled in and connected.* The two big problems I'd see would be (1) Social Order and (2) Internet Connectivity. DC's are not fortresses, and internet backbone fibers/routers/etc. are distributed & kinda fragile. *After all the large-scale power outages & near-outages of recent decades, Cloudflare has no excuse if they lack really-good backup generators at critical facilities. And with their size, Cloudflare must support enough "critical during major disaster" internet services to actually get such generators.
ZiiS 3y ago
And most of thier SREs. Spending 30 hours to recover from the worst natural disaster in recorded history is slightly diffrent then from a ground fault on a single transformer.
fanf2 3y ago
There’s also the Portland Hills and Gales Creek fault zones to worry about.

ahoka 3y ago

They would then go into very detailed description how tectonic activity caused the outage.