4 ms·

In this case it is not. A power outage in a critical data-center is the root cause here: https://www.cloudflarestatus.com/incidents/hm7491k53ppg https://www.clo

by nkcmr 3y ago

In this case it is not. A power outage in a critical data-center is the root cause here: https://www.cloudflarestatus.com/incidents/hm7491k53ppg https://www.cloudflarestatus.com/incidents/hm7491k53ppg

ta1243 3y ago
If you can't cope with the loss of a data centre you're not really running a resilient system.
- itslennysfault 3y ago
  On yesterday's post someone that used to work at CF mentioned that PDX is "the brain" and if it goes down data stops propagating and starts getting stale. It's crazy to me that a company that is so critical to SO MUCH of the traffic on the internet doesn't even have a failover strategy for "the brain" of their operation.
  - sangnoir 3y ago
    Not great to know an aspiring hyperscaler is one tornado away from insolvency.
    - seized 3y ago
      I dont think PDX would get many tornados BUT the PNW has the joys of earthquakes that are overdue.
      - ta1243 3y ago
        It's amusing, I have plans on how much service degradation I can accept and what I need to keep working in the UK if the Thames Barrier is breached (which would flood most of the UK's internet connectivity in places like Telehouse, Sovereign House, etc) To have 30% of the internet relying on a single building in a single city is hilarious.
- dogweather 3y ago
  Especially 15 hours after the DC comes back up and you're still offline.
tux3 3y ago
What I'm confused by is we had "power partially restored" 22 hours ago, and no news from PDX02 since. I assume both Clouflare and Flexential are on DEFCON 1 right now, but I'm wondering if it might be more than just the building going dark. There's something about a failover than was attempted and crashed halfway through, but unclear if that's what's causing the 24h+ situation.
- throwaway_13140 3y ago
  if it was a state sponsored attack, various 3 letter agencies may want to investigate which could slow down restoration of service
sponaugle 3y ago
I am in this DC, and we lost power to all of our racks but one. Power was restored about 2 hours later. I would assume Cloudflare had some significant failures in equipment due to the power drop. We lost a couple of servers that didn't come back up, which is not uncommon problem with hardware that has been running without a power-off for 4-5 years.

by nkcmr 3y ago

ta1243 3y ago

If you can't cope with the loss of a data centre you're not really running a resilient system.

itslennysfault 3y ago
On yesterday's post someone that used to work at CF mentioned that PDX is "the brain" and if it goes down data stops propagating and starts getting stale. It's crazy to me that a company that is so critical to SO MUCH of the traffic on the internet doesn't even have a failover strategy for "the brain" of their operation.
- sangnoir 3y ago
  Not great to know an aspiring hyperscaler is one tornado away from insolvency.
  - seized 3y ago
    I dont think PDX would get many tornados BUT the PNW has the joys of earthquakes that are overdue.
    - ta1243 3y ago
      It's amusing, I have plans on how much service degradation I can accept and what I need to keep working in the UK if the Thames Barrier is breached (which would flood most of the UK's internet connectivity in places like Telehouse, Sovereign House, etc) To have 30% of the internet relying on a single building in a single city is hilarious.
dogweather 3y ago
Especially 15 hours after the DC comes back up and you're still offline.

tux3 3y ago

What I'm confused by is we had "power partially restored" 22 hours ago, and no news from PDX02 since. I assume both Clouflare and Flexential are on DEFCON 1 right now, but I'm wondering if it might be more than just the building going dark. There's something about a failover than was attempted and crashed halfway through, but unclear if that's what's causing the 24h+ situation.

throwaway_13140 3y ago
if it was a state sponsored attack, various 3 letter agencies may want to investigate which could slow down restoration of service

sponaugle 3y ago

I am in this DC, and we lost power to all of our racks but one. Power was restored about 2 hours later. I would assume Cloudflare had some significant failures in equipment due to the power drop. We lost a couple of servers that didn't come back up, which is not uncommon problem with hardware that has been running without a power-off for 4-5 years.