4 ms·

Power outage seems really odd. Don't datacenters usually have multiple redundant power supplies + on-site backup power generation? Maybe power "incident" mean s

by nialv7 2y ago

Power outage seems really odd. Don't datacenters usually have multiple redundant power supplies + on-site backup power generation? Maybe power "incident" mean something else?

tgsovlerkhgsel 2y ago
Things happen, e.g. the redundant system also failing, the system that should handle the failover failing, a short circuit that causes enough chaos that the redundant supply shuts down rather than feeding power into a potential fault, ...
- toomuchtodo 2y ago
  Amazon famously had an outage that was generator gremlins root cause [1] and tries to engineer around it [2]. [1] https://news.ycombinator.com/item?id=38114946 https://news.ycombinator.com/item?id=38114946 [2] https://news.ycombinator.com/item?id=15676820 https://news.ycombinator.com/item?id=15676820
jasonjayr 2y ago
I had equipment at a colo facility -- that had a (licensed, bonded, not fly-by-night) electrical tech accidentally drop a tool into the main bus connecting mains, generators, and batteries. They are lucky they were able to walk away, but the facility was dark till someone could get in there and give the power equipment the green light.
__turbobrew__ 2y ago
The switching equipment can fail. Had this happen at a DC where the switching equipment arc flashed when going from mains to diesel generators. The switching equipment detected the arc and then locked out until someone onsite could inspect the equipment and override it. The rack UPSes only lasted like 5 minutes and then everything went dark.
- emptiestplace 2y ago
  That sounds quiet.
  - crest 2y ago
    The sound of silence can be deafening.
    - Daneel_ 2y ago
      The other end of the spectrum is just as terrifying - CRAC unit failure. I never knew fans could scream so loudly.
- red-iron-pine 2y ago
  this happened to a dupont fabros facility in northern va in I wanna say ~2012? derecho storms hammered the area and killed power. external power lines in failed, and the ATS hung or died when switching to the N+1 diesel generators. since it never got switched to diesel, the UPS systems kept things going for the standard interval (e.g. ~3-5 minutes) and then ran out of power, and then everything went down. AWS died and IIRC it took a lot of stuff with it, most notably reddit, etc.
mattlondon 2y ago
Sometimes redundant things are also knocked out, e.g. https://www.thestack.technology/google-cloud-paris-water-fire-europe-west9/ https://www.thestack.technology/google-cloud-paris-water-fir... Tl;Dr fire in one data center hall was put out with water, water leaked into other hall's power generator and battery area. Turns out loads of water and power generation equipment don't mix well, and servers don't like sitting in puddles of water. Another one was where the whole building burnt down rather than "just" the power equipment. https://www.datacenterdynamics.com/en/news/fire-destroys-ovhclouds-sbg2-data-center-strasbourg/ https://www.datacenterdynamics.com/en/news/fire-destroys-ovh...
eps 2y ago
Back in the 00's there was a power outage in downtown Vancouver. It caused Peer1 to fall back to their generators... that weren't tested for ages. They struggled for 5 minutes, gave up and bursted into flames, resulting in the colo not being able to go back online even when the main power was restored. That was an epic mess. Especially considering they positioned themselves as the most technically sophisticated colo in the region. So, yeah, it happens.

by nialv7 2y ago

Power outage seems really odd. Don't datacenters usually have multiple redundant power supplies + on-site backup power generation? Maybe power "incident" mean something else?

tgsovlerkhgsel 2y ago

Things happen, e.g. the redundant system also failing, the system that should handle the failover failing, a short circuit that causes enough chaos that the redundant supply shuts down rather than feeding power into a potential fault, ...

toomuchtodo 2y ago
Amazon famously had an outage that was generator gremlins root cause [1] and tries to engineer around it [2]. [1] https://news.ycombinator.com/item?id=38114946 https://news.ycombinator.com/item?id=38114946 [2] https://news.ycombinator.com/item?id=15676820 https://news.ycombinator.com/item?id=15676820

jasonjayr 2y ago

I had equipment at a colo facility -- that had a (licensed, bonded, not fly-by-night) electrical tech accidentally drop a tool into the main bus connecting mains, generators, and batteries. They are lucky they were able to walk away, but the facility was dark till someone could get in there and give the power equipment the green light.

__turbobrew__ 2y ago

The switching equipment can fail. Had this happen at a DC where the switching equipment arc flashed when going from mains to diesel generators. The switching equipment detected the arc and then locked out until someone onsite could inspect the equipment and override it. The rack UPSes only lasted like 5 minutes and then everything went dark.

emptiestplace 2y ago
That sounds quiet.
- crest 2y ago
  The sound of silence can be deafening.
  - Daneel_ 2y ago
    The other end of the spectrum is just as terrifying - CRAC unit failure. I never knew fans could scream so loudly.
red-iron-pine 2y ago
this happened to a dupont fabros facility in northern va in I wanna say ~2012? derecho storms hammered the area and killed power. external power lines in failed, and the ATS hung or died when switching to the N+1 diesel generators. since it never got switched to diesel, the UPS systems kept things going for the standard interval (e.g. ~3-5 minutes) and then ran out of power, and then everything went down. AWS died and IIRC it took a lot of stuff with it, most notably reddit, etc.

mattlondon 2y ago

Sometimes redundant things are also knocked out, e.g. https://www.thestack.technology/google-cloud-paris-water-fire-europe-west9/ https://www.thestack.technology/google-cloud-paris-water-fir... Tl;Dr fire in one data center hall was put out with water, water leaked into other hall's power generator and battery area. Turns out loads of water and power generation equipment don't mix well, and servers don't like sitting in puddles of water. Another one was where the whole building burnt down rather than "just" the power equipment. https://www.datacenterdynamics.com/en/news/fire-destroys-ovhclouds-sbg2-data-center-strasbourg/ https://www.datacenterdynamics.com/en/news/fire-destroys-ovh...

eps 2y ago

Back in the 00's there was a power outage in downtown Vancouver. It caused Peer1 to fall back to their generators... that weren't tested for ages. They struggled for 5 minutes, gave up and bursted into flames, resulting in the colo not being able to go back online even when the main power was restored. That was an epic mess. Especially considering they positioned themselves as the most technically sophisticated colo in the region. So, yeah, it happens.