3 ms·

> there was no way to fail back power without an outage, then them going off and building their own equipment. Anywhere I can read about that?

michaelt 3y ago
Maybe [1] which is about per-rack UPSes, to reduce the blast radius of UPS failure? Pretty sensible IMHO - I live in a country with a reliable electricity grid, and outages due to UPS malfunction are about as common as power outages. [1] https://www.datacenterdynamics.com/en/news/aws-develops-its-own-rack-ups/ https://www.datacenterdynamics.com/en/news/aws-develops-its-...
- nyrikki 3y ago
  In large data centers, rack level UPSs are impractical for many reasons like cost and efficiency, but the big problem is that modern power densities are so high that you want rows to fail if cooling isn't available. It doesn't take long without cooling to cook equipment to the point of failure or reduced reliability. 7 to 16kW per rack is common even in these older colo facilities. And there never would have been enough UPS to make up for not enough replacement breakers on site.
  - michaelt 3y ago
    But isn't the UPS only expected to last for 15 minutes or so, to give the backup generators time to start up? Or to perform a fast-but-graceful migration when the generator doesn't start up? I thought most DCs just pause the cooling until the generator comes up, rather than running the cooling on battery power?
    - throw0101d 3y ago
      > But isn't the UPS only expected to last for 15 minutes or so, to give the backup generators time to start up? They are expected to last that long, but if the batteries are on year 4 of their 5-year life, that may not happen. What also may not happen is the generator starting up. Or the automatic transfer switch (ATS) not working properly: it should be on either input Feed A or Feed B, but when it tries to throw itself over (making a loud kah-chunk sound), it gets stuck in between—this happened to us once. "The perversity of the universe always tends toward a maximum." — Finagle's Law of Dynamic Negatives
      - jabart 3y ago
        I know our DC does load test on the batteries and generators. There are also 5 generators on site. I get all the emails about these tests since they are marked low risk. Fun story, small IT company had a natural gas generator installed at the new office they built out. Power went out for the first time, lights go out, generator kicks on, lights turn on, lights turn off. Long story short the electrician didn't have the right breaker size in the truck and used a small one. Building was empty when it was tested which is why bigger DCs have a load generator to test at load.
  - jeffbee 3y ago
    Google has been using rack-level UPS for many years, so I'm curious to hear why you think that is impractical specifically in large datacenters. https://cloud.google.com/blog/products/compute/google-joins-open-compute-project-to-drive-standards-in-it-infrastructure#:~:text=Our%2048V%20architecture%20has%20since,the%20design%20and%20its%20reliability https://cloud.google.com/blog/products/compute/google-joins-....
    - justsomehnguy 3y ago
      Your typical company, even if it has hundreds of racks, usually at the scale and workloads needed at Google, where you can plug off the racks and not really notice it.
    - lathiat 3y ago
      Probably because Google are designing the entire hardware (its not a Dell rack server) and data center together, they are also responsible for the entire workload if something goes wrong and likely also have data center redundancy for an entire power loss for the entire workload there. In third party data centers you're co-located with other customers that have mostly comodity server hardware which won't have those options, they generally (at least in my experience) all provide UPS power as part of the facilitiy so you have it already and something like a fire if it did happen, would impact lots of other customers who likely don't have redunancy between data centers or fire compartments. Additionally depending on your exact setup (it's possible you may have dark fibre directly out of the facilitiy, but often not) you're likely to be uplinked through powered equipment at the same facility, that wouldn't have the extra UPS power either. Plus as others mentioned, the loss of cooling is the biggest problem. In fairness maybe that would buy you a few minutes for the kindof automatic switchover they talked about. But that's quite a bit of co-ordination to have your own power, to not rely on anyone else being powered and then to be able to co-ordinate your shutdown before the room gets too hot - and not be at risk of overheating the room for others. Plus there are safety issues like emergency power out for fire fighters etc which again, I'm sure Google could deal with at a facility-wide level, but if you're a small fish in a much bigger data center its harder to co-ordinate that kindof thing. So while I'm sure it's all possible, there are obvious headwinds in many directions.
  - throw0101d 3y ago
    > It doesn't take long without cooling to cook equipment to the point of failure or reduced reliability. IIRC, it's about 30 minutes—irrelevant of room size. This is because while smaller rooms have less air volume as a 'buffer', they also you can't fit much gear. While huge data centres have huge volumes of air, they also have lots of gear. So there's a coincidental proportional relationship what you (as a rule of thumb) have 30 minutes before things cook, regardless of room size. (At least for 'general purpose' computing: GPUs are probably another matter.)
- Dylan16807 3y ago
  You normally reduce the blast radius of UPS failures by having two supplies for each server. So I don't think that's it. The link someone else put about overriding faults might be it, if OP misremembered the problem.
  - michaelt 3y ago
    Well in principle, sure. And yet here we are, reading an article about "a total loss of power [...] following a reportedly simultaneous failure of four [...] switchboards serving all of Cloudflare’s cages. This meant both primary and redundant power paths were deactivated across the entire environment."
    - Dylan16807 3y ago
      I'm not sure what you mean. Wouldn't we have the same level of outage with per-rack UPSes?
- mysteria 3y ago
  > Pretty sensible IMHO - I live in a country with a reliable electricity grid, and outages due to UPS malfunction are about as common as power outages. Hence why modern servers and their associated networking equipment have dual power supplies which could be connected to two seperate UPS systems. It would be very unlikely to have them both fail at once. In a less important home/small business scenario typically one supply is connected to the UPS and the other is connected to the wall via a surge protector.
- jabart 3y ago
  Per-Rack UPS is not allowed in our local Flexential DC. Reason being is that in case of a fire, the whole room needs to go dark on power for fire control. We do have two redundant AC circuits on two different breakers. But our DC was from a company that got bought out by Flexential so maybe, hopefully, its setup different.
  - iJohnDoe 3y ago
    It’s a good idea not to have per rack UPS in a colo. Customers are really bad at maintaining equipment Making sure customers replace batteries every five years might be difficult. Also, in my opinion, all UPS manufactures make horrible UPS equipment. There can be so many different types of failures or problems or glitches. You don’t want a UPS leaking or exploding. Also, there are laws in some states that only allow so many pounds of batteries in a building.
    - EricE 3y ago
      I can confirm that the vast majority of UPS's are garbage. I happily pay more to source Eaton whenever I can.
hx833001 3y ago
https://perspectives.mvdirona.com/2017/04/at-scale-rare-events-arent-rare/ https://perspectives.mvdirona.com/2017/04/at-scale-rare-even...

michaelt 3y ago

Maybe [1] which is about per-rack UPSes, to reduce the blast radius of UPS failure? Pretty sensible IMHO - I live in a country with a reliable electricity grid, and outages due to UPS malfunction are about as common as power outages. [1] https://www.datacenterdynamics.com/en/news/aws-develops-its-own-rack-ups/ https://www.datacenterdynamics.com/en/news/aws-develops-its-...

nyrikki 3y ago
In large data centers, rack level UPSs are impractical for many reasons like cost and efficiency, but the big problem is that modern power densities are so high that you want rows to fail if cooling isn't available. It doesn't take long without cooling to cook equipment to the point of failure or reduced reliability. 7 to 16kW per rack is common even in these older colo facilities. And there never would have been enough UPS to make up for not enough replacement breakers on site.
- michaelt 3y ago
  But isn't the UPS only expected to last for 15 minutes or so, to give the backup generators time to start up? Or to perform a fast-but-graceful migration when the generator doesn't start up? I thought most DCs just pause the cooling until the generator comes up, rather than running the cooling on battery power?
  - throw0101d 3y ago
    > But isn't the UPS only expected to last for 15 minutes or so, to give the backup generators time to start up? They are expected to last that long, but if the batteries are on year 4 of their 5-year life, that may not happen. What also may not happen is the generator starting up. Or the automatic transfer switch (ATS) not working properly: it should be on either input Feed A or Feed B, but when it tries to throw itself over (making a loud kah-chunk sound), it gets stuck in between—this happened to us once. "The perversity of the universe always tends toward a maximum." — Finagle's Law of Dynamic Negatives
    - jabart 3y ago
      I know our DC does load test on the batteries and generators. There are also 5 generators on site. I get all the emails about these tests since they are marked low risk. Fun story, small IT company had a natural gas generator installed at the new office they built out. Power went out for the first time, lights go out, generator kicks on, lights turn on, lights turn off. Long story short the electrician didn't have the right breaker size in the truck and used a small one. Building was empty when it was tested which is why bigger DCs have a load generator to test at load.
- jeffbee 3y ago
  Google has been using rack-level UPS for many years, so I'm curious to hear why you think that is impractical specifically in large datacenters. https://cloud.google.com/blog/products/compute/google-joins-open-compute-project-to-drive-standards-in-it-infrastructure#:~:text=Our%2048V%20architecture%20has%20since,the%20design%20and%20its%20reliability https://cloud.google.com/blog/products/compute/google-joins-....
  - justsomehnguy 3y ago
    Your typical company, even if it has hundreds of racks, usually at the scale and workloads needed at Google, where you can plug off the racks and not really notice it.
  - lathiat 3y ago
    Probably because Google are designing the entire hardware (its not a Dell rack server) and data center together, they are also responsible for the entire workload if something goes wrong and likely also have data center redundancy for an entire power loss for the entire workload there. In third party data centers you're co-located with other customers that have mostly comodity server hardware which won't have those options, they generally (at least in my experience) all provide UPS power as part of the facilitiy so you have it already and something like a fire if it did happen, would impact lots of other customers who likely don't have redunancy between data centers or fire compartments. Additionally depending on your exact setup (it's possible you may have dark fibre directly out of the facilitiy, but often not) you're likely to be uplinked through powered equipment at the same facility, that wouldn't have the extra UPS power either. Plus as others mentioned, the loss of cooling is the biggest problem. In fairness maybe that would buy you a few minutes for the kindof automatic switchover they talked about. But that's quite a bit of co-ordination to have your own power, to not rely on anyone else being powered and then to be able to co-ordinate your shutdown before the room gets too hot - and not be at risk of overheating the room for others. Plus there are safety issues like emergency power out for fire fighters etc which again, I'm sure Google could deal with at a facility-wide level, but if you're a small fish in a much bigger data center its harder to co-ordinate that kindof thing. So while I'm sure it's all possible, there are obvious headwinds in many directions.
- throw0101d 3y ago
  > It doesn't take long without cooling to cook equipment to the point of failure or reduced reliability. IIRC, it's about 30 minutes—irrelevant of room size. This is because while smaller rooms have less air volume as a 'buffer', they also you can't fit much gear. While huge data centres have huge volumes of air, they also have lots of gear. So there's a coincidental proportional relationship what you (as a rule of thumb) have 30 minutes before things cook, regardless of room size. (At least for 'general purpose' computing: GPUs are probably another matter.)
Dylan16807 3y ago
You normally reduce the blast radius of UPS failures by having two supplies for each server. So I don't think that's it. The link someone else put about overriding faults might be it, if OP misremembered the problem.
- michaelt 3y ago
  Well in principle, sure. And yet here we are, reading an article about "a total loss of power [...] following a reportedly simultaneous failure of four [...] switchboards serving all of Cloudflare’s cages. This meant both primary and redundant power paths were deactivated across the entire environment."
  - Dylan16807 3y ago
    I'm not sure what you mean. Wouldn't we have the same level of outage with per-rack UPSes?
mysteria 3y ago
> Pretty sensible IMHO - I live in a country with a reliable electricity grid, and outages due to UPS malfunction are about as common as power outages. Hence why modern servers and their associated networking equipment have dual power supplies which could be connected to two seperate UPS systems. It would be very unlikely to have them both fail at once. In a less important home/small business scenario typically one supply is connected to the UPS and the other is connected to the wall via a surge protector.
jabart 3y ago
Per-Rack UPS is not allowed in our local Flexential DC. Reason being is that in case of a fire, the whole room needs to go dark on power for fire control. We do have two redundant AC circuits on two different breakers. But our DC was from a company that got bought out by Flexential so maybe, hopefully, its setup different.
- iJohnDoe 3y ago
  It’s a good idea not to have per rack UPS in a colo. Customers are really bad at maintaining equipment Making sure customers replace batteries every five years might be difficult. Also, in my opinion, all UPS manufactures make horrible UPS equipment. There can be so many different types of failures or problems or glitches. You don’t want a UPS leaking or exploding. Also, there are laws in some states that only allow so many pounds of batteries in a building.
  - EricE 3y ago
    I can confirm that the vast majority of UPS's are garbage. I happily pay more to source Eaton whenever I can.

hx833001 3y ago

https://perspectives.mvdirona.com/2017/04/at-scale-rare-events-arent-rare/ https://perspectives.mvdirona.com/2017/04/at-scale-rare-even...