6 ms·

It always made me wonder why liquid cooling wasn't more of a thing for datacenters. Water has a massive amount of thermal capacity and can quickly and in bulk

It always made me wonder why liquid cooling wasn't more of a thing for datacenters. Water has a massive amount of thermal capacity and can quickly and in bulk be cooled to optimal temperatures. You'd probably still need fans and AC to dissipate heat of non-liquid cooled parts, but for the big energy items like CPUs and GPUs/compute engines, you could ship out huge amounts of heat fairly quickly and directly. I guess the complexity and risk of a leak would be a problem, but for amazon sized data centers that doesn't seem like a major concern.

lub 3y ago
OVH prominently uses water cooling including custom components with their own design. https://blog.ovhcloud.com/water-cooling-from-innovation-to-disruption-part-i/ https://blog.ovhcloud.com/water-cooling-from-innovation-to-d... https://blog.ovhcloud.com/water-cooling-from-innovation-to-disruption-part-ii/ https://blog.ovhcloud.com/water-cooling-from-innovation-to-d... https://blog.ovhcloud.com/new-hybrid-immersion-liquid-cooling-developments-at-ovhcloud/ https://blog.ovhcloud.com/new-hybrid-immersion-liquid-coolin...
adev_ 3y ago
> It always made me wonder why liquid cooling wasn't more of a thing for datacenters. Liquid cooling is almost a defacto-standard in data centers in the HPC world. The Top of the TOP500 machines are all liquid cooled. Not by choice, but due to physics constraints. There is a big gap in power density between the HPC world and the usual datacenter-commodity-hardware world. Commodity DS are designed with the assumption that the average machine will run with a fraction of it's maximum load. HPC systems at the opposite are designed to operate safely at 100% load all the time. In a previous company where I worked, we attempted to install a medium size HPC cluster in a well-known commerical datacenter and network provider. The commercial of the DS almost felt from his chair when we announced the power requirements.
- bayindirh 3y ago
  > we attempted to install a medium size HPC cluster in a well-known commerical Datacenter and network provider. The commercial of the DS almost fall from his chair when we announced the power requirements. Heh. We tried it too. They didn’t believe that a single node used their entire rack’s budget at first.
  - KolenCh 3y ago
    I'm going through a similar thing. Next week our procurement is arriving, but most likely majority of the nodes will sit in a box for a while before they can figure out how to power them... Also, this involves a UK university where it would takes forever to upgrade the power delivery to the building/floor/room. There's no planned upgrade whatsoever so people just need to make do with what we have.
  - mrgaro 3y ago
    Sounds fascinating. Can you give any more details? What kind of nodes are they and how they differ from "traditional" DC hardware, say from Supermicro?
    - lhoff 3y ago
      The difference is GPUs. A normal dual socket system serving a database or webserver use under medium load around 200-300W, One of these [1] equipped with 10xA100 can easily use in the ballpark of 3kW under load. So we are talking 10x the power usage. [1]https://www.supermicro.com/en/products/system/gpu/5u/sys-521ge-tnrt https://www.supermicro.com/en/products/system/gpu/5u/sys-521...
      - mrgaro 3y ago
        Thanks, this makes sense!
- deleted 3y ago
  [deleted]
- KolenCh 3y ago
  I’m a bit skeptical about the claim that the top of the TOP500 are all liquid cooled due to physical constraints. (Where the “physics” here seems to mean the density and property of air cool in general but ignoring the environment of the machine such as the ambient weather.) The one data point I know well is NERSC, and has been air cooled until relatively recently. Part of the success of air cool in the past was the great Bay Area weather that is typically cooled enough. I forgot the exact reason for upgrading to water cool, but it was before the recent upgrade to a new machine (Perlmutter). It may have to do with our weather getting more extreme so that there are more incidences that air cool only will “throttle” your machine. The reason I’m skeptical about that claim is that the ambient temperature also play a role. But certainly the density is increasing so may be the biggest supercomputers at the moment are too dense (in power consumption) to be air cooled.
  - adev_ 3y ago
    If you take the top5 of the top500 (01/2024): - Frontier: Liquid cooled - Aurora: liquid cooled - Eagle: liquid cooled - Fugaku: liquid cooled - Lumi: Liquid cooled with heat recycling. The need for liquid cooled arrived with the usage of accelerators that bumped significantly the power density. That said, Fugaku which is not accelerator driven is also liquid cooled today. I am surprised to learn that NERSC was air cooled until recently. Most of the BGQ machines, that were dominating the top500 10 years ago, were already liquid cooled.
    - KolenCh 3y ago
      I probably wasn't clear. I don't doubt your first statement "The Top of the TOP500 machines are all liquid cooled." My skepticism is your second statement "due to physics constraints". It may very well be true but I'm just skeptical that it can't be done with a combination of engineering and ambient environment (mostly temperature but also a tradeoff between density and space). May be I focused on the word physics too much (being a physicist), but it seems the decision would basically be a cost-benefit analysis and risk management which involves many factors including money, maturity of solutions in the market, safety, etc. For example, at NERSC, there are real risk of a massive earthquake long overdue (the whole floor is quake-proof, but I guess the risk is reduced, not eliminated), so I guess they probably have considered this in that design choice. But perhaps you're right that the physics is the ultimate factor here: what comes in must goes out. With order of magnitude scale of increase in power coming in, water seems to be the most effective and cheap entity to absorb those heat and be transported outside the floor very efficiently.
victotronics 3y ago
Immersion cooling is getting big. At the last Supercomputing conference I probably saw at least a dozen vendors of immersion cooling equipment. My datacenter has one cluster with liquid cooling caps over the sockets, and two immersed clusters. The latter two have basins of various degrees of sophistication under them for when they do spring a leak.
bayindirh 3y ago
Because it’s complex. Even more complex than “engineered” air. You need two circuits, and a CDU between them. Coolants needs maintaining. You add antifreeze, biocides, etc. Air is brute force. It cools everything it touches. Liquid cooling is serialized in a node. Two sockets? Second will be hotter. HBA not making good contact? It’ll overheat. You add extensive leak detection subsystems, the amount of coolant moving in your primary circuit becomes massive. Currently you can remove 97% of the heat via liquid (including the PSUs), and it’s cheaper to do so than air, but it’s not “rails, screws, cables, power on”. Air cooled systems can be turned on in a week. Liquid cooled ones take a month. However, using liquid is mandatory after some point. You can’t cool systems that dense and under that load with air. They’ll melt.
- gopher_space 3y ago
  What's this all look like without an atmosphere?
  - eutropia 3y ago
    Worse, heat dissipation is a major constraint for spacecraft and satellites because you can only radiate heat away as infrared photons.
    - gopher_space 3y ago
      Couldn't you heat up and poop out blocks of material all day long, or is that starting down a "solution creates more heat than it dissipates" path?
    - taneq 3y ago
      Doesn’t have to be infrared but yeah, space isn’t “cold” so much as it’s an insulator.
      - _a_a_a_ 3y ago
        at ~2 kelvin I'd have thought you can radiate away a truckload of heat surely
_kb 3y ago
In contexts where there's a good chance of standardisation, I believe it is. Both OCP [0] and Open19 [1] have liquid cooling as part of the standard. [0]: https://www.opencompute.org/projects/cooling-environments https://www.opencompute.org/projects/cooling-environments [1]: https://gitlab.com/open19/v2-specification/-/blob/main/system_architecture_public.pdf https://gitlab.com/open19/v2-specification/-/blob/main/syste...
hinkley 3y ago
I wonder if you could design a rack or half rack that has a connector for attaching a minisplit refrigerant line directly to it, instead of cooling the entire room. Problem is that’s how you sell 1-3 racks not an entire room of them. Ethylene lines might be more practical, easier to wire into existing systems.

lub 3y ago

OVH prominently uses water cooling including custom components with their own design. https://blog.ovhcloud.com/water-cooling-from-innovation-to-disruption-part-i/ https://blog.ovhcloud.com/water-cooling-from-innovation-to-d... https://blog.ovhcloud.com/water-cooling-from-innovation-to-disruption-part-ii/ https://blog.ovhcloud.com/water-cooling-from-innovation-to-d... https://blog.ovhcloud.com/new-hybrid-immersion-liquid-cooling-developments-at-ovhcloud/ https://blog.ovhcloud.com/new-hybrid-immersion-liquid-coolin...

adev_ 3y ago

> It always made me wonder why liquid cooling wasn't more of a thing for datacenters. Liquid cooling is almost a defacto-standard in data centers in the HPC world. The Top of the TOP500 machines are all liquid cooled. Not by choice, but due to physics constraints. There is a big gap in power density between the HPC world and the usual datacenter-commodity-hardware world. Commodity DS are designed with the assumption that the average machine will run with a fraction of it's maximum load. HPC systems at the opposite are designed to operate safely at 100% load all the time. In a previous company where I worked, we attempted to install a medium size HPC cluster in a well-known commerical datacenter and network provider. The commercial of the DS almost felt from his chair when we announced the power requirements.

bayindirh 3y ago
> we attempted to install a medium size HPC cluster in a well-known commerical Datacenter and network provider. The commercial of the DS almost fall from his chair when we announced the power requirements. Heh. We tried it too. They didn’t believe that a single node used their entire rack’s budget at first.
- KolenCh 3y ago
  I'm going through a similar thing. Next week our procurement is arriving, but most likely majority of the nodes will sit in a box for a while before they can figure out how to power them... Also, this involves a UK university where it would takes forever to upgrade the power delivery to the building/floor/room. There's no planned upgrade whatsoever so people just need to make do with what we have.
- mrgaro 3y ago
  Sounds fascinating. Can you give any more details? What kind of nodes are they and how they differ from "traditional" DC hardware, say from Supermicro?
  - lhoff 3y ago
    The difference is GPUs. A normal dual socket system serving a database or webserver use under medium load around 200-300W, One of these [1] equipped with 10xA100 can easily use in the ballpark of 3kW under load. So we are talking 10x the power usage. [1]https://www.supermicro.com/en/products/system/gpu/5u/sys-521ge-tnrt https://www.supermicro.com/en/products/system/gpu/5u/sys-521...
    - mrgaro 3y ago
      Thanks, this makes sense!
deleted 3y ago
[deleted]
KolenCh 3y ago
I’m a bit skeptical about the claim that the top of the TOP500 are all liquid cooled due to physical constraints. (Where the “physics” here seems to mean the density and property of air cool in general but ignoring the environment of the machine such as the ambient weather.) The one data point I know well is NERSC, and has been air cooled until relatively recently. Part of the success of air cool in the past was the great Bay Area weather that is typically cooled enough. I forgot the exact reason for upgrading to water cool, but it was before the recent upgrade to a new machine (Perlmutter). It may have to do with our weather getting more extreme so that there are more incidences that air cool only will “throttle” your machine. The reason I’m skeptical about that claim is that the ambient temperature also play a role. But certainly the density is increasing so may be the biggest supercomputers at the moment are too dense (in power consumption) to be air cooled.
- adev_ 3y ago
  If you take the top5 of the top500 (01/2024): - Frontier: Liquid cooled - Aurora: liquid cooled - Eagle: liquid cooled - Fugaku: liquid cooled - Lumi: Liquid cooled with heat recycling. The need for liquid cooled arrived with the usage of accelerators that bumped significantly the power density. That said, Fugaku which is not accelerator driven is also liquid cooled today. I am surprised to learn that NERSC was air cooled until recently. Most of the BGQ machines, that were dominating the top500 10 years ago, were already liquid cooled.
  - KolenCh 3y ago
    I probably wasn't clear. I don't doubt your first statement "The Top of the TOP500 machines are all liquid cooled." My skepticism is your second statement "due to physics constraints". It may very well be true but I'm just skeptical that it can't be done with a combination of engineering and ambient environment (mostly temperature but also a tradeoff between density and space). May be I focused on the word physics too much (being a physicist), but it seems the decision would basically be a cost-benefit analysis and risk management which involves many factors including money, maturity of solutions in the market, safety, etc. For example, at NERSC, there are real risk of a massive earthquake long overdue (the whole floor is quake-proof, but I guess the risk is reduced, not eliminated), so I guess they probably have considered this in that design choice. But perhaps you're right that the physics is the ultimate factor here: what comes in must goes out. With order of magnitude scale of increase in power coming in, water seems to be the most effective and cheap entity to absorb those heat and be transported outside the floor very efficiently.

victotronics 3y ago

Immersion cooling is getting big. At the last Supercomputing conference I probably saw at least a dozen vendors of immersion cooling equipment. My datacenter has one cluster with liquid cooling caps over the sockets, and two immersed clusters. The latter two have basins of various degrees of sophistication under them for when they do spring a leak.

bayindirh 3y ago

Because it’s complex. Even more complex than “engineered” air. You need two circuits, and a CDU between them. Coolants needs maintaining. You add antifreeze, biocides, etc. Air is brute force. It cools everything it touches. Liquid cooling is serialized in a node. Two sockets? Second will be hotter. HBA not making good contact? It’ll overheat. You add extensive leak detection subsystems, the amount of coolant moving in your primary circuit becomes massive. Currently you can remove 97% of the heat via liquid (including the PSUs), and it’s cheaper to do so than air, but it’s not “rails, screws, cables, power on”. Air cooled systems can be turned on in a week. Liquid cooled ones take a month. However, using liquid is mandatory after some point. You can’t cool systems that dense and under that load with air. They’ll melt.

gopher_space 3y ago
What's this all look like without an atmosphere?
- eutropia 3y ago
  Worse, heat dissipation is a major constraint for spacecraft and satellites because you can only radiate heat away as infrared photons.
  - gopher_space 3y ago
    Couldn't you heat up and poop out blocks of material all day long, or is that starting down a "solution creates more heat than it dissipates" path?
  - taneq 3y ago
    Doesn’t have to be infrared but yeah, space isn’t “cold” so much as it’s an insulator.
    - _a_a_a_ 3y ago
      at ~2 kelvin I'd have thought you can radiate away a truckload of heat surely

_kb 3y ago

In contexts where there's a good chance of standardisation, I believe it is. Both OCP [0] and Open19 [1] have liquid cooling as part of the standard. [0]: https://www.opencompute.org/projects/cooling-environments https://www.opencompute.org/projects/cooling-environments [1]: https://gitlab.com/open19/v2-specification/-/blob/main/system_architecture_public.pdf https://gitlab.com/open19/v2-specification/-/blob/main/syste...

hinkley 3y ago

I wonder if you could design a rack or half rack that has a connector for attaching a minisplit refrigerant line directly to it, instead of cooling the entire room. Problem is that’s how you sell 1-3 racks not an entire room of them. Ethylene lines might be more practical, easier to wire into existing systems.