7 ms·

This is quite interesting as they claim their datacenter design does better than Uptime's Tier3+ design requirements which require redundant power supply paths.

This is quite interesting as they claim their datacenter design does better than Uptime's Tier3+ design requirements which require redundant power supply paths. [https://aws.amazon.com/compliance/uptimeinstitute/ https://aws.amazon.com/compliance/uptimeinstitute/]. I really hope they publish a thorough RCA for this incident.

tyingq 5y ago
"Electrical power systems are designed to be fully redundant so that in the event of a disruption, uninterruptible power supply units can be engaged for certain functions, while generators can provide backup power for the entire facility." https://aws.amazon.com/compliance/data-center/infrastructure-layer/ https://aws.amazon.com/compliance/data-center/infrastructure... So they have 2 different sources of power coming in. And generators. They do mention the UPS is only for "certain functions", so I guess it's not enough to handle full load while generators spin up if the 2 primaries go out. Or perhaps some failure in the source switching equipment (typically called a "static transfer switch"). Some detail on different approaches: https://www.donwil.com/wp-content/uploads/white-papers/Using-Static-Transfer-Switches-to-Enhance-Data-Center-Availability-and-Maintainability-Technical-Note-TN-00011.pdf https://www.donwil.com/wp-content/uploads/white-papers/Using...
- vinay_ys 5y ago
  Usually when someone claims T3+ they mean they have UPS clusters in 3+1 (or such) configuration and two different such UPS clusters power two power-strips in a rack. Then, would also have incoming grid power supply from two different HV sub-stations with non-intersecting cable paths. They would also have diesel power generators in 3+1 or 5+2 configurations with automatic startup time in seconds. The UPS's power storage (chemical or potential energy based devices) can hold enough energy to handle full load for several minutes. If these are design and maintained correctly, even while concurrent scheduled maintenance is ongoing, an unexpected component failure should not cause catastrophic outage. At each layer (grid incomers, generator incomers, UPS power incomers) there are switches to switch over whenever there's a need (maintenance or failure). If they claim tier4, then they basically have everything in n+n configuration.
  - tyingq 5y ago
    Though that doesn't match very well with "uninterruptible power supply units can be engaged for certain functions". It sounds worded to convey that the UPS is limited in some way. An interesting old summary of their 2012 us-east-1 incident with power/generators/ups/switching: https://aws.amazon.com/message/67457/ https://aws.amazon.com/message/67457/
- dylan604 5y ago
  The generators should be powering up as soon as one of the 2 different sources goes down. It takes generators a few minutes to power up and get "warmed up". If they don't start this process until both mains sources are down, then oops, there's power outage. I used to work next door to a "major" cable TV station's broadcast location. They had multiple generators on-site, and one of them was running 24/7 (they rotated which one was hot). A major power outage hit, and there was a thunderous roar as all of the generators fired up. The channel never went off the air.
  - hughesjj 5y ago
    I thought running a generator full time was illegal AF due to environmental regulations?
    - AndyJames 5y ago
      Not really. If there's no local law against it then it's legal especially outside the cities.
  - tyingq 5y ago
    There are setups where the UPS is designed to last long enough for generator spin up as well. I believe it's the most common setup if you have both. I assume spinning up the generators for very short-lived line power blips might be undesirable.
    - greyhair 5y ago
      I was in a Bell Labs facility that had notoriously bad power. We occupied the building before the second main feed had been fully approved by the state and run to the building. Our main computer lab had a serial UPS that was online 100% of the time, though the inverters where under a very light load. If the mains even acted 'weird' (dips, bad power factor, spikes) the UPS jumped full on, and didn't revert to main power until the main was stable for some duration of time. The UPS was able to carry the full lab (which was quite large) for about two hours, allowing plenty of time for the generator to fire up. The UPS ran a lot, and because the main was 'weird', the outages were often short, the generator wouldn't even start during the first ten minutes of UPS coverage. Of course, the rest of the building would be dark, other then emergency lighting. I was a embedded firmware engineer, and our development lab was directly on the wall behind the UPS. When it fired into 100% mode, it roared, mostly from cooling. It was sort of a heads up that the power was likely to fail soon.
- AtlasBarfed 5y ago
  Has datacenter power redundancy undergone any sort of revolution with grid storage becoming industrial scale? I wonder if a lot of AWS dc design in this area predates the battery grid storage revolution with (what my impression is) a far faster adaptation/switchover time than a generator spin up, and possibly software systems that work to detect and switch over quickly? AWS can claim it will be best of breed, but they aren't going to throw out a DC power redundancy investment (or threaten downtime) that they can't wring more ROI on.
  - tyingq 5y ago
    I'd be surprised. Data centers eat a lot of energy, and it's hard to beat the energy density of diesel (120 MJ/kg vs ~1 for batteries) and the ability to have nearby tanks or scheduled trucks. Tesla apparently did some early pilot stuff: https://www.datacenterdynamics.com/en/analysis/teslas-powerpack-proposes-battery-power-for-data-centers/ https://www.datacenterdynamics.com/en/analysis/teslas-powerp...
  - Gravityloss 5y ago
    When deciding data center locations, companies certainly have in mind the quality of the electrical infrastructure in that country or region... Maybe it could affect people buying services as well.
- rainbowzootsuit 5y ago
  Likely the UPS can't run HVAC, and you are in an overheat condition in about two minutes with a fully loaded data center without cooling. Proportionately longer as load is reduced.
JshWright 5y ago
> I really hope they publish a thorough RCA for this incident. We're still waiting on the RCA for last week's us-west outage...

tyingq 5y ago

"Electrical power systems are designed to be fully redundant so that in the event of a disruption, uninterruptible power supply units can be engaged for certain functions, while generators can provide backup power for the entire facility." https://aws.amazon.com/compliance/data-center/infrastructure-layer/ https://aws.amazon.com/compliance/data-center/infrastructure... So they have 2 different sources of power coming in. And generators. They do mention the UPS is only for "certain functions", so I guess it's not enough to handle full load while generators spin up if the 2 primaries go out. Or perhaps some failure in the source switching equipment (typically called a "static transfer switch"). Some detail on different approaches: https://www.donwil.com/wp-content/uploads/white-papers/Using-Static-Transfer-Switches-to-Enhance-Data-Center-Availability-and-Maintainability-Technical-Note-TN-00011.pdf https://www.donwil.com/wp-content/uploads/white-papers/Using...

vinay_ys 5y ago
Usually when someone claims T3+ they mean they have UPS clusters in 3+1 (or such) configuration and two different such UPS clusters power two power-strips in a rack. Then, would also have incoming grid power supply from two different HV sub-stations with non-intersecting cable paths. They would also have diesel power generators in 3+1 or 5+2 configurations with automatic startup time in seconds. The UPS's power storage (chemical or potential energy based devices) can hold enough energy to handle full load for several minutes. If these are design and maintained correctly, even while concurrent scheduled maintenance is ongoing, an unexpected component failure should not cause catastrophic outage. At each layer (grid incomers, generator incomers, UPS power incomers) there are switches to switch over whenever there's a need (maintenance or failure). If they claim tier4, then they basically have everything in n+n configuration.
- tyingq 5y ago
  Though that doesn't match very well with "uninterruptible power supply units can be engaged for certain functions". It sounds worded to convey that the UPS is limited in some way. An interesting old summary of their 2012 us-east-1 incident with power/generators/ups/switching: https://aws.amazon.com/message/67457/ https://aws.amazon.com/message/67457/
dylan604 5y ago
The generators should be powering up as soon as one of the 2 different sources goes down. It takes generators a few minutes to power up and get "warmed up". If they don't start this process until both mains sources are down, then oops, there's power outage. I used to work next door to a "major" cable TV station's broadcast location. They had multiple generators on-site, and one of them was running 24/7 (they rotated which one was hot). A major power outage hit, and there was a thunderous roar as all of the generators fired up. The channel never went off the air.
- hughesjj 5y ago
  I thought running a generator full time was illegal AF due to environmental regulations?
  - AndyJames 5y ago
    Not really. If there's no local law against it then it's legal especially outside the cities.
- tyingq 5y ago
  There are setups where the UPS is designed to last long enough for generator spin up as well. I believe it's the most common setup if you have both. I assume spinning up the generators for very short-lived line power blips might be undesirable.
  - greyhair 5y ago
    I was in a Bell Labs facility that had notoriously bad power. We occupied the building before the second main feed had been fully approved by the state and run to the building. Our main computer lab had a serial UPS that was online 100% of the time, though the inverters where under a very light load. If the mains even acted 'weird' (dips, bad power factor, spikes) the UPS jumped full on, and didn't revert to main power until the main was stable for some duration of time. The UPS was able to carry the full lab (which was quite large) for about two hours, allowing plenty of time for the generator to fire up. The UPS ran a lot, and because the main was 'weird', the outages were often short, the generator wouldn't even start during the first ten minutes of UPS coverage. Of course, the rest of the building would be dark, other then emergency lighting. I was a embedded firmware engineer, and our development lab was directly on the wall behind the UPS. When it fired into 100% mode, it roared, mostly from cooling. It was sort of a heads up that the power was likely to fail soon.
AtlasBarfed 5y ago
Has datacenter power redundancy undergone any sort of revolution with grid storage becoming industrial scale? I wonder if a lot of AWS dc design in this area predates the battery grid storage revolution with (what my impression is) a far faster adaptation/switchover time than a generator spin up, and possibly software systems that work to detect and switch over quickly? AWS can claim it will be best of breed, but they aren't going to throw out a DC power redundancy investment (or threaten downtime) that they can't wring more ROI on.
- tyingq 5y ago
  I'd be surprised. Data centers eat a lot of energy, and it's hard to beat the energy density of diesel (120 MJ/kg vs ~1 for batteries) and the ability to have nearby tanks or scheduled trucks. Tesla apparently did some early pilot stuff: https://www.datacenterdynamics.com/en/analysis/teslas-powerpack-proposes-battery-power-for-data-centers/ https://www.datacenterdynamics.com/en/analysis/teslas-powerp...
- Gravityloss 5y ago
  When deciding data center locations, companies certainly have in mind the quality of the electrical infrastructure in that country or region... Maybe it could affect people buying services as well.
rainbowzootsuit 5y ago
Likely the UPS can't run HVAC, and you are in an overheat condition in about two minutes with a fully loaded data center without cooling. Proportionately longer as load is reduced.

JshWright 5y ago

> I really hope they publish a thorough RCA for this incident. We're still waiting on the RCA for last week's us-west outage...