4 ms·

> I've only had a DC fail once when the engineer was performing work on the power circuitry for the DC and thought he was taking down one, but was in fact the w

> I've only had a DC fail once when the engineer was performing work on the power circuitry for the DC and thought he was taking down one, but was in fact the wrong one and took both power circuits down at the same time. This is all local scale. Your setup would not survive a data center scale power outage. At scale power outages are datacenter scale. Data centers lose supply lines. They lose transformers. Sometimes they lose primary feed and secondary feed at the same time. Automatic transfer switches cannot be tested periodically i.e. they are typically tested once. Testing them is not "fire up a generator and see if we can draw from it" It is cheaper to design a system that must be up which accounts for a data center being totally down and a portion of the system being totally unavailable than to add more datacenter mitigations.

vel0city 5y ago
The only full datacenter outage I've personally experienced was a power maintenance tech testing the transfer switch between systems where the power was 90 degrees out of phase. Big oof.
ItsBob 5y ago
Yes but if you have reliable power from two different sources then the biggest risk (I'd imagine) is the failover circuitry! Something that should be tested tbh. Also, there are banks of batteries and generators in between the power company cables and the kit: did they not kick-in? Again, this is all pure speculation: I have absolutely no idea of the exact failure, nor how their infrastructure is held together - this is all just speculation for the hell of it :)
- merlyn 5y ago
  Frying hardware can affect much wider scope. I've had bad power supplies fry out taking the whole power circuit with it, and thus half (or whatever fraction) of the rack's power. I've also had bad power supplies bring down the whole machine as they shunted everything internal too. When things go bad, anything can happen. You can provide the best effort, and it'll usually work as expected, but there will always be something that can overcome your best efforts.
- notyourday 5y ago
  > Yes but if you have reliable power from two different sources then the biggest risk (I'd imagine) is the failover circuitry! Something that should be tested tbh. That's ATS. It is not really advisable to test their under load performance because the failure of an ATS would be catastrophic. ATS typically would be tested at the installation and after that their parameters would be monitored. Replacing a functional in line ATS would be a 9-12 months long project. > Also, there are banks of batteries and generators in between the power company cables and the kit: did they not kick-in? At high energy you are pretty much always going to use an ATS.
  - belfalas 5y ago
    > the failure of an ATS would be catastrophic Because that would mean no power at all to the DC and no way to get it back? (I am completely ignorant on this topic)
    - notyourday 5y ago
      > Because that would mean no power at all to the DC and no way to get it back? (I am completely ignorant on this topic) While most of smarts in the ATS are in the electronics, the really nasty failures come from the mechanical part. At the end of the day a high energy ATS looks just like a switch behind a meter in your house. There's a lip that goes from one position to another, except in a high energy ATS the lip is big and when the transfer occurs it slams from one source to another. There are only so many of those physical slams that it can withstand to being with so you want to minimize that number. The second failure mode is that after transfer to non-main source, the lip can get stuck there, making it impossible to switch back on the main. [Once I have seem the lip melt into the secondary position. While I thought it was weird, the guys from the power company said it is not that uncommon.] This creates a massive problem as the non-main source is typically not designed for long term 24x7 operation. So now you are stuck on a secondary feeding system and you cant just transfer to main without de-energizing the system i.e. taking the power out of the entire data center.
bombcar 5y ago
The datacenter we were in had dual-sourced grid power (two separate grid connections on opposite sides of the block, coming from different substations) along with a room of batteries (good for iirc 1hr total runtime for the whole datacenter, setup in quad banks, two on each "rail"), _and_ multiple independent massive diesel generators, which they ran and switched power to every month for at least an hour. And to top it off each rack had its own smaller UPS at the bottom and top, fed off both rails, and each server was fed from both. We never had a power issue there; in fact SDGE would ask them to throw to the generators during potential brown-out conditions. Of course this was a datacenter that was a former General Atomics setup iirc ...
- notyourday 5y ago
  We were in a triple sourced data center. Fed by three different substations. Everything worked like a charm. Until Sandy hit. It did not affect us at all. But it affected the power company. And everything still worked fine, until one of the transfer switches transferred into UPS position and stopped working in that position.
ClumsyPilot 5y ago
"It is cheaper to design a system that must be up which accounts for a data center being totally down and a portion of the system being totally unavailable than to add more datacenter mitigations." Citation needed - the same issue with testing, data races and expensive bandwidth come up.
- notyourday 5y ago
  At high energy the lead time for the components is measured not in days but in years.
  - ClumsyPilot 5y ago
    And so is development time of any distributed software system, and training time required to operate it correctly
    - notyourday 5y ago
      > And so is development time of any distributed software system, and training time required to operate it correctly Software is much easier than hardware. If you are to start a project today in this kind of hardware, you will be operating it in 2029, without changes.
      - ClumsyPilot 5y ago
        "Software is much easier than hardware. If you are to start a project today in this kind of hardware, you will be operating it in 2029, without changes." I don't think this makes sense, you are using the three statements "Software is cheaper", "Software takes less time" and "software is easier" as if they all mean the same thing, and proving one means proving all of them. Hardware takes a long time, okay, that does not mean it's expensive. Building a hydroelectric dam takes 20 years, but it provides the cheapest source of electricity that ever existed. Ships can take a decade from order to delivery, they are the cheapest mode of transport.
theideaofcoffee 5y ago
Transfer switches at any facility that's worth being colocated in are exercised as periodically as the generators to which they connect. In all of the facilities I have had systems in (>20MW total steady state IT load), that meant once per month at minimum to keep generators happy -and to ensure the transfer functionality works-, and more often if the local grid demands it, e.g. ComEd in Chicago, or Dominion in NoVA asking for load shedding.

vel0city 5y ago

The only full datacenter outage I've personally experienced was a power maintenance tech testing the transfer switch between systems where the power was 90 degrees out of phase. Big oof.

ItsBob 5y ago

Yes but if you have reliable power from two different sources then the biggest risk (I'd imagine) is the failover circuitry! Something that should be tested tbh. Also, there are banks of batteries and generators in between the power company cables and the kit: did they not kick-in? Again, this is all pure speculation: I have absolutely no idea of the exact failure, nor how their infrastructure is held together - this is all just speculation for the hell of it :)

merlyn 5y ago
Frying hardware can affect much wider scope. I've had bad power supplies fry out taking the whole power circuit with it, and thus half (or whatever fraction) of the rack's power. I've also had bad power supplies bring down the whole machine as they shunted everything internal too. When things go bad, anything can happen. You can provide the best effort, and it'll usually work as expected, but there will always be something that can overcome your best efforts.
notyourday 5y ago
> Yes but if you have reliable power from two different sources then the biggest risk (I'd imagine) is the failover circuitry! Something that should be tested tbh. That's ATS. It is not really advisable to test their under load performance because the failure of an ATS would be catastrophic. ATS typically would be tested at the installation and after that their parameters would be monitored. Replacing a functional in line ATS would be a 9-12 months long project. > Also, there are banks of batteries and generators in between the power company cables and the kit: did they not kick-in? At high energy you are pretty much always going to use an ATS.
- belfalas 5y ago
  > the failure of an ATS would be catastrophic Because that would mean no power at all to the DC and no way to get it back? (I am completely ignorant on this topic)
  - notyourday 5y ago
    > Because that would mean no power at all to the DC and no way to get it back? (I am completely ignorant on this topic) While most of smarts in the ATS are in the electronics, the really nasty failures come from the mechanical part. At the end of the day a high energy ATS looks just like a switch behind a meter in your house. There's a lip that goes from one position to another, except in a high energy ATS the lip is big and when the transfer occurs it slams from one source to another. There are only so many of those physical slams that it can withstand to being with so you want to minimize that number. The second failure mode is that after transfer to non-main source, the lip can get stuck there, making it impossible to switch back on the main. [Once I have seem the lip melt into the secondary position. While I thought it was weird, the guys from the power company said it is not that uncommon.] This creates a massive problem as the non-main source is typically not designed for long term 24x7 operation. So now you are stuck on a secondary feeding system and you cant just transfer to main without de-energizing the system i.e. taking the power out of the entire data center.

bombcar 5y ago

The datacenter we were in had dual-sourced grid power (two separate grid connections on opposite sides of the block, coming from different substations) along with a room of batteries (good for iirc 1hr total runtime for the whole datacenter, setup in quad banks, two on each "rail"), _and_ multiple independent massive diesel generators, which they ran and switched power to every month for at least an hour. And to top it off each rack had its own smaller UPS at the bottom and top, fed off both rails, and each server was fed from both. We never had a power issue there; in fact SDGE would ask them to throw to the generators during potential brown-out conditions. Of course this was a datacenter that was a former General Atomics setup iirc ...

notyourday 5y ago
We were in a triple sourced data center. Fed by three different substations. Everything worked like a charm. Until Sandy hit. It did not affect us at all. But it affected the power company. And everything still worked fine, until one of the transfer switches transferred into UPS position and stopped working in that position.

ClumsyPilot 5y ago

"It is cheaper to design a system that must be up which accounts for a data center being totally down and a portion of the system being totally unavailable than to add more datacenter mitigations." Citation needed - the same issue with testing, data races and expensive bandwidth come up.

notyourday 5y ago
At high energy the lead time for the components is measured not in days but in years.
- ClumsyPilot 5y ago
  And so is development time of any distributed software system, and training time required to operate it correctly
  - notyourday 5y ago
    > And so is development time of any distributed software system, and training time required to operate it correctly Software is much easier than hardware. If you are to start a project today in this kind of hardware, you will be operating it in 2029, without changes.
    - ClumsyPilot 5y ago
      "Software is much easier than hardware. If you are to start a project today in this kind of hardware, you will be operating it in 2029, without changes." I don't think this makes sense, you are using the three statements "Software is cheaper", "Software takes less time" and "software is easier" as if they all mean the same thing, and proving one means proving all of them. Hardware takes a long time, okay, that does not mean it's expensive. Building a hydroelectric dam takes 20 years, but it provides the cheapest source of electricity that ever existed. Ships can take a decade from order to delivery, they are the cheapest mode of transport.

theideaofcoffee 5y ago

Transfer switches at any facility that's worth being colocated in are exercised as periodically as the generators to which they connect. In all of the facilities I have had systems in (>20MW total steady state IT load), that meant once per month at minimum to keep generators happy -and to ensure the transfer functionality works-, and more often if the local grid demands it, e.g. ComEd in Chicago, or Dominion in NoVA asking for load shedding.