3 ms·

In large data centers, rack level UPSs are impractical for many reasons like cost and efficiency, but the big problem is that modern power densities are so high

In large data centers, rack level UPSs are impractical for many reasons like cost and efficiency, but the big problem is that modern power densities are so high that you want rows to fail if cooling isn't available. It doesn't take long without cooling to cook equipment to the point of failure or reduced reliability. 7 to 16kW per rack is common even in these older colo facilities. And there never would have been enough UPS to make up for not enough replacement breakers on site.

michaelt 3y ago
But isn't the UPS only expected to last for 15 minutes or so, to give the backup generators time to start up? Or to perform a fast-but-graceful migration when the generator doesn't start up? I thought most DCs just pause the cooling until the generator comes up, rather than running the cooling on battery power?
- throw0101d 3y ago
  > But isn't the UPS only expected to last for 15 minutes or so, to give the backup generators time to start up? They are expected to last that long, but if the batteries are on year 4 of their 5-year life, that may not happen. What also may not happen is the generator starting up. Or the automatic transfer switch (ATS) not working properly: it should be on either input Feed A or Feed B, but when it tries to throw itself over (making a loud kah-chunk sound), it gets stuck in between—this happened to us once. "The perversity of the universe always tends toward a maximum." — Finagle's Law of Dynamic Negatives
  - jabart 3y ago
    I know our DC does load test on the batteries and generators. There are also 5 generators on site. I get all the emails about these tests since they are marked low risk. Fun story, small IT company had a natural gas generator installed at the new office they built out. Power went out for the first time, lights go out, generator kicks on, lights turn on, lights turn off. Long story short the electrician didn't have the right breaker size in the truck and used a small one. Building was empty when it was tested which is why bigger DCs have a load generator to test at load.
jeffbee 3y ago
Google has been using rack-level UPS for many years, so I'm curious to hear why you think that is impractical specifically in large datacenters. https://cloud.google.com/blog/products/compute/google-joins-open-compute-project-to-drive-standards-in-it-infrastructure#:~:text=Our%2048V%20architecture%20has%20since,the%20design%20and%20its%20reliability https://cloud.google.com/blog/products/compute/google-joins-....
- justsomehnguy 3y ago
  Your typical company, even if it has hundreds of racks, usually at the scale and workloads needed at Google, where you can plug off the racks and not really notice it.
- lathiat 3y ago
  Probably because Google are designing the entire hardware (its not a Dell rack server) and data center together, they are also responsible for the entire workload if something goes wrong and likely also have data center redundancy for an entire power loss for the entire workload there. In third party data centers you're co-located with other customers that have mostly comodity server hardware which won't have those options, they generally (at least in my experience) all provide UPS power as part of the facilitiy so you have it already and something like a fire if it did happen, would impact lots of other customers who likely don't have redunancy between data centers or fire compartments. Additionally depending on your exact setup (it's possible you may have dark fibre directly out of the facilitiy, but often not) you're likely to be uplinked through powered equipment at the same facility, that wouldn't have the extra UPS power either. Plus as others mentioned, the loss of cooling is the biggest problem. In fairness maybe that would buy you a few minutes for the kindof automatic switchover they talked about. But that's quite a bit of co-ordination to have your own power, to not rely on anyone else being powered and then to be able to co-ordinate your shutdown before the room gets too hot - and not be at risk of overheating the room for others. Plus there are safety issues like emergency power out for fire fighters etc which again, I'm sure Google could deal with at a facility-wide level, but if you're a small fish in a much bigger data center its harder to co-ordinate that kindof thing. So while I'm sure it's all possible, there are obvious headwinds in many directions.
throw0101d 3y ago
> It doesn't take long without cooling to cook equipment to the point of failure or reduced reliability. IIRC, it's about 30 minutes—irrelevant of room size. This is because while smaller rooms have less air volume as a 'buffer', they also you can't fit much gear. While huge data centres have huge volumes of air, they also have lots of gear. So there's a coincidental proportional relationship what you (as a rule of thumb) have 30 minutes before things cook, regardless of room size. (At least for 'general purpose' computing: GPUs are probably another matter.)

michaelt 3y ago

But isn't the UPS only expected to last for 15 minutes or so, to give the backup generators time to start up? Or to perform a fast-but-graceful migration when the generator doesn't start up? I thought most DCs just pause the cooling until the generator comes up, rather than running the cooling on battery power?

throw0101d 3y ago
> But isn't the UPS only expected to last for 15 minutes or so, to give the backup generators time to start up? They are expected to last that long, but if the batteries are on year 4 of their 5-year life, that may not happen. What also may not happen is the generator starting up. Or the automatic transfer switch (ATS) not working properly: it should be on either input Feed A or Feed B, but when it tries to throw itself over (making a loud kah-chunk sound), it gets stuck in between—this happened to us once. "The perversity of the universe always tends toward a maximum." — Finagle's Law of Dynamic Negatives
- jabart 3y ago
  I know our DC does load test on the batteries and generators. There are also 5 generators on site. I get all the emails about these tests since they are marked low risk. Fun story, small IT company had a natural gas generator installed at the new office they built out. Power went out for the first time, lights go out, generator kicks on, lights turn on, lights turn off. Long story short the electrician didn't have the right breaker size in the truck and used a small one. Building was empty when it was tested which is why bigger DCs have a load generator to test at load.

jeffbee 3y ago

Google has been using rack-level UPS for many years, so I'm curious to hear why you think that is impractical specifically in large datacenters. https://cloud.google.com/blog/products/compute/google-joins-open-compute-project-to-drive-standards-in-it-infrastructure#:~:text=Our%2048V%20architecture%20has%20since,the%20design%20and%20its%20reliability https://cloud.google.com/blog/products/compute/google-joins-....

justsomehnguy 3y ago
Your typical company, even if it has hundreds of racks, usually at the scale and workloads needed at Google, where you can plug off the racks and not really notice it.
lathiat 3y ago
Probably because Google are designing the entire hardware (its not a Dell rack server) and data center together, they are also responsible for the entire workload if something goes wrong and likely also have data center redundancy for an entire power loss for the entire workload there. In third party data centers you're co-located with other customers that have mostly comodity server hardware which won't have those options, they generally (at least in my experience) all provide UPS power as part of the facilitiy so you have it already and something like a fire if it did happen, would impact lots of other customers who likely don't have redunancy between data centers or fire compartments. Additionally depending on your exact setup (it's possible you may have dark fibre directly out of the facilitiy, but often not) you're likely to be uplinked through powered equipment at the same facility, that wouldn't have the extra UPS power either. Plus as others mentioned, the loss of cooling is the biggest problem. In fairness maybe that would buy you a few minutes for the kindof automatic switchover they talked about. But that's quite a bit of co-ordination to have your own power, to not rely on anyone else being powered and then to be able to co-ordinate your shutdown before the room gets too hot - and not be at risk of overheating the room for others. Plus there are safety issues like emergency power out for fire fighters etc which again, I'm sure Google could deal with at a facility-wide level, but if you're a small fish in a much bigger data center its harder to co-ordinate that kindof thing. So while I'm sure it's all possible, there are obvious headwinds in many directions.

throw0101d 3y ago

> It doesn't take long without cooling to cook equipment to the point of failure or reduced reliability. IIRC, it's about 30 minutes—irrelevant of room size. This is because while smaller rooms have less air volume as a 'buffer', they also you can't fit much gear. While huge data centres have huge volumes of air, they also have lots of gear. So there's a coincidental proportional relationship what you (as a rule of thumb) have 30 minutes before things cook, regardless of room size. (At least for 'general purpose' computing: GPUs are probably another matter.)