5 ms·

Status page is updated now: https://azure.status.microsoft/ https://azure.status.microsoft/ >Impact Statement: Starting at 18:44 UTC on 26 Dec 2024, you have b

Status page is updated now: https://azure.status.microsoft/ https://azure.status.microsoft/ >Impact Statement: Starting at 18:44 UTC on 26 Dec 2024, you have been identified as a customer who was impacted by a power incident in South Central US and may experience a degraded experience. >Current Status: There was a power incident in the South Central US AZ03 which affected multiple services. We have applied mitigation and are actively validating recovery to the impacted services. Further updates will be provided in 60 minutes, or sooner as events warrant. The times are the same for OpenAI - first notice from 11:00 PST (19:00 UTC)

nialv7 2y ago
Power outage seems really odd. Don't datacenters usually have multiple redundant power supplies + on-site backup power generation? Maybe power "incident" mean something else?
- tgsovlerkhgsel 2y ago
  Things happen, e.g. the redundant system also failing, the system that should handle the failover failing, a short circuit that causes enough chaos that the redundant supply shuts down rather than feeding power into a potential fault, ...
  - toomuchtodo 2y ago
    Amazon famously had an outage that was generator gremlins root cause [1] and tries to engineer around it [2]. [1] https://news.ycombinator.com/item?id=38114946 https://news.ycombinator.com/item?id=38114946 [2] https://news.ycombinator.com/item?id=15676820 https://news.ycombinator.com/item?id=15676820
- jasonjayr 2y ago
  I had equipment at a colo facility -- that had a (licensed, bonded, not fly-by-night) electrical tech accidentally drop a tool into the main bus connecting mains, generators, and batteries. They are lucky they were able to walk away, but the facility was dark till someone could get in there and give the power equipment the green light.
- __turbobrew__ 2y ago
  The switching equipment can fail. Had this happen at a DC where the switching equipment arc flashed when going from mains to diesel generators. The switching equipment detected the arc and then locked out until someone onsite could inspect the equipment and override it. The rack UPSes only lasted like 5 minutes and then everything went dark.
  - emptiestplace 2y ago
    That sounds quiet.
    - crest 2y ago
      The sound of silence can be deafening.
      - Daneel_ 2y ago
        The other end of the spectrum is just as terrifying - CRAC unit failure. I never knew fans could scream so loudly.
  - red-iron-pine 2y ago
    this happened to a dupont fabros facility in northern va in I wanna say ~2012? derecho storms hammered the area and killed power. external power lines in failed, and the ATS hung or died when switching to the N+1 diesel generators. since it never got switched to diesel, the UPS systems kept things going for the standard interval (e.g. ~3-5 minutes) and then ran out of power, and then everything went down. AWS died and IIRC it took a lot of stuff with it, most notably reddit, etc.
- mattlondon 2y ago
  Sometimes redundant things are also knocked out, e.g. https://www.thestack.technology/google-cloud-paris-water-fire-europe-west9/ https://www.thestack.technology/google-cloud-paris-water-fir... Tl;Dr fire in one data center hall was put out with water, water leaked into other hall's power generator and battery area. Turns out loads of water and power generation equipment don't mix well, and servers don't like sitting in puddles of water. Another one was where the whole building burnt down rather than "just" the power equipment. https://www.datacenterdynamics.com/en/news/fire-destroys-ovhclouds-sbg2-data-center-strasbourg/ https://www.datacenterdynamics.com/en/news/fire-destroys-ovh...
- eps 2y ago
  Back in the 00's there was a power outage in downtown Vancouver. It caused Peer1 to fall back to their generators... that weren't tested for ages. They struggled for 5 minutes, gave up and bursted into flames, resulting in the colo not being able to go back online even when the main power was restored. That was an epic mess. Especially considering they positioned themselves as the most technically sophisticated colo in the region. So, yeah, it happens.
Y_Y 2y ago
> you ... may experience a degraded experience What an eloquent way to say "our service isn't working".
- caseyy 2y ago
  It’s the same accountability shirking language as when layoffs “have affected you” instead of “I mismanaged this business and as a result I’m firing you”. It’s always the same abstract invisible hand that just keeps affecting everyone! Scott Alexander’s Moloch perhaps :)
  - Y_Y 2y ago
    > Scott Alexander’s Moloch perhaps :) I almost wish I hadn't read that blog post because now I see Moloch and and his invisible handprints everywhere. Anyway here it is: https://slatestarcodex.com/2014/07/30/meditations-on-moloch/ https://slatestarcodex.com/2014/07/30/meditations-on-moloch/
- orlp 2y ago
  That's the next stage of deflection after passive voice. 1. In normal and honest language you state things you have done, and their consequences. 2. Passive voice attempts to deflect blame by only stating the consequences as if they just magically occurred through divine intervention. 3. In the next stage they don't even acknowledge the consequences, and instead place the entire issue inside your experience of the facts.
  - perching_aix 2y ago
    I don't think this stuff is the work of the devil personally by a long shot. We don't know what exactly caused the power issue and they might not have had a root cause at the time either. Let's assume that their power redundancy equipment failed, say, due to insufficient maintenance. This is not an active action, it's a passive one (they didn't do their maintenance duties properly and now it blew). So there is nothing to say for point #1 and #2. There's also the part where they say that the customers they identified as impacted may be experiencing a service degradation. This may sound pedantic, but I think it is not an entirely unreasonable phrasing. Maybe my business isn't actively relying on the resources I have deployed in that datacenter. How would they know (#3)? Should I clean those resources up? Possibly. Depends on my access patterns and other considerations. It reads like face (and ass) saving legal esque language. But there's a reason face and ass saving legalese sounds like it does.
amir734jj 2y ago
There was a major power outage in the South Carolina data center, and I was on call for that. It's resolved now.
- sunaookami 2y ago
  Thank you for your service!
- VirusNewbie 2y ago
  Unsung heroes keeping the internet running. Thank you for your service!

nialv7 2y ago

Power outage seems really odd. Don't datacenters usually have multiple redundant power supplies + on-site backup power generation? Maybe power "incident" mean something else?

tgsovlerkhgsel 2y ago
Things happen, e.g. the redundant system also failing, the system that should handle the failover failing, a short circuit that causes enough chaos that the redundant supply shuts down rather than feeding power into a potential fault, ...
- toomuchtodo 2y ago
  Amazon famously had an outage that was generator gremlins root cause [1] and tries to engineer around it [2]. [1] https://news.ycombinator.com/item?id=38114946 https://news.ycombinator.com/item?id=38114946 [2] https://news.ycombinator.com/item?id=15676820 https://news.ycombinator.com/item?id=15676820
jasonjayr 2y ago
I had equipment at a colo facility -- that had a (licensed, bonded, not fly-by-night) electrical tech accidentally drop a tool into the main bus connecting mains, generators, and batteries. They are lucky they were able to walk away, but the facility was dark till someone could get in there and give the power equipment the green light.
__turbobrew__ 2y ago
The switching equipment can fail. Had this happen at a DC where the switching equipment arc flashed when going from mains to diesel generators. The switching equipment detected the arc and then locked out until someone onsite could inspect the equipment and override it. The rack UPSes only lasted like 5 minutes and then everything went dark.
- emptiestplace 2y ago
  That sounds quiet.
  - crest 2y ago
    The sound of silence can be deafening.
    - Daneel_ 2y ago
      The other end of the spectrum is just as terrifying - CRAC unit failure. I never knew fans could scream so loudly.
- red-iron-pine 2y ago
  this happened to a dupont fabros facility in northern va in I wanna say ~2012? derecho storms hammered the area and killed power. external power lines in failed, and the ATS hung or died when switching to the N+1 diesel generators. since it never got switched to diesel, the UPS systems kept things going for the standard interval (e.g. ~3-5 minutes) and then ran out of power, and then everything went down. AWS died and IIRC it took a lot of stuff with it, most notably reddit, etc.
mattlondon 2y ago
Sometimes redundant things are also knocked out, e.g. https://www.thestack.technology/google-cloud-paris-water-fire-europe-west9/ https://www.thestack.technology/google-cloud-paris-water-fir... Tl;Dr fire in one data center hall was put out with water, water leaked into other hall's power generator and battery area. Turns out loads of water and power generation equipment don't mix well, and servers don't like sitting in puddles of water. Another one was where the whole building burnt down rather than "just" the power equipment. https://www.datacenterdynamics.com/en/news/fire-destroys-ovhclouds-sbg2-data-center-strasbourg/ https://www.datacenterdynamics.com/en/news/fire-destroys-ovh...
eps 2y ago
Back in the 00's there was a power outage in downtown Vancouver. It caused Peer1 to fall back to their generators... that weren't tested for ages. They struggled for 5 minutes, gave up and bursted into flames, resulting in the colo not being able to go back online even when the main power was restored. That was an epic mess. Especially considering they positioned themselves as the most technically sophisticated colo in the region. So, yeah, it happens.

Y_Y 2y ago

> you ... may experience a degraded experience What an eloquent way to say "our service isn't working".

caseyy 2y ago
It’s the same accountability shirking language as when layoffs “have affected you” instead of “I mismanaged this business and as a result I’m firing you”. It’s always the same abstract invisible hand that just keeps affecting everyone! Scott Alexander’s Moloch perhaps :)
- Y_Y 2y ago
  > Scott Alexander’s Moloch perhaps :) I almost wish I hadn't read that blog post because now I see Moloch and and his invisible handprints everywhere. Anyway here it is: https://slatestarcodex.com/2014/07/30/meditations-on-moloch/ https://slatestarcodex.com/2014/07/30/meditations-on-moloch/
orlp 2y ago
That's the next stage of deflection after passive voice. 1. In normal and honest language you state things you have done, and their consequences. 2. Passive voice attempts to deflect blame by only stating the consequences as if they just magically occurred through divine intervention. 3. In the next stage they don't even acknowledge the consequences, and instead place the entire issue inside your experience of the facts.
- perching_aix 2y ago
  I don't think this stuff is the work of the devil personally by a long shot. We don't know what exactly caused the power issue and they might not have had a root cause at the time either. Let's assume that their power redundancy equipment failed, say, due to insufficient maintenance. This is not an active action, it's a passive one (they didn't do their maintenance duties properly and now it blew). So there is nothing to say for point #1 and #2. There's also the part where they say that the customers they identified as impacted may be experiencing a service degradation. This may sound pedantic, but I think it is not an entirely unreasonable phrasing. Maybe my business isn't actively relying on the resources I have deployed in that datacenter. How would they know (#3)? Should I clean those resources up? Possibly. Depends on my access patterns and other considerations. It reads like face (and ass) saving legal esque language. But there's a reason face and ass saving legalese sounds like it does.

amir734jj 2y ago

There was a major power outage in the South Carolina data center, and I was on call for that. It's resolved now.

sunaookami 2y ago
Thank you for your service!
VirusNewbie 2y ago
Unsung heroes keeping the internet running. Thank you for your service!