8 ms·

Google's Liquid Cooling

betaby 1y ago
Relevant. OVH has been using liquid cooling for many years: https://blog.ovhcloud.com/new-hybrid-immersion-liquid-cooling-developments-at-ovhcloud/ https://blog.ovhcloud.com/new-hybrid-immersion-liquid-coolin... https://www.youtube.com/watch?v=S_DHJcQpumI https://www.youtube.com/watch?v=S_DHJcQpumI
- jeffbee 1y ago
  And yet their claimed PUE is 1.26 which is pretty bad. One way to characterize that overall PUE figure is they waste 3x as much on overhead as Google (claimed 1.09 global PUE) or Meta (1.08).
  - Loic 1y ago
    10 years ago, OVH datacenter SBG1 was below 1.1 with a mixture of natural convection + liquid cooling. But I have not been following in the latest years.
- bri3d 1y ago
  Your linked articles are about immersion cooling, which is "liquid cooling," I suppose, but a very different thing. Do OVH actually use immersion cooling in production? This seems like a "labs" product that hasn't been fully baked yet. OVH _do_ definitely use traditional water-loop/water-block liquid cooling like Google are presenting, described here: https://blog.ovhcloud.com/understanding-ovhclouds-data-centres-how-delta-t-helps-with-water-efficiency/ https://blog.ovhcloud.com/understanding-ovhclouds-data-centr... and visually in https://www.youtube.com/watch?v=RFzirpvTiOo https://www.youtube.com/watch?v=RFzirpvTiOo , but while it looks very similar architecturally their setup seems almost comically inefficient compared to Google's according to their PUE disclosures.
  - TiredOfLife 1y ago
    Based on how their data centers burn down, they don't even use liquid in their fire suppression systems.
michaelt 1y ago
> TPU chips are hooked up in series in the loop, which naturally means some chips will get hotter liquid that has already passed other chips in the loop. Cooling capacity is budgeted based on the requirements of the last chip in each loop. Of course, it's worth noting that if you've got four chips, each putting out 250W of power, and a pump pushing 1 litres of water per minute through them, water at the outlet must be 14°C hotter than water at the inlet, because of the specific heat capacity of water. That's true whether the water flows through the chips in series, or in parallel.
- foota 1y ago
  Hm... but in the case when the chips are in serial, the heat transfer from the last chip will be less than when the chips are in parallel, because the rate of heating is proportional to the difference in temperature, and the water starts at a lower temperature for the parallel case for this last theoretical chip.
  - fraserphysics 1y ago
    One way to characterize the cost of cooling is entropy production. As you say, cooling is proportional to difference in temperature. However, entropy production is also proportional to temperature difference. It's not my field, but it looks like an interesting challenge to optimize competing objectives.
  - 0x457 1y ago
    Yes, but water is constantly moving in a loop. It's not like you use water to cool chip #1, and then it moves to chip #2, it's constantly moving, so temperature delta isn't that much.
    - smachiz 1y ago
      in their first serial design, that's exactly what it was doing.
      - Dylan16807 1y ago
jonathaneunice 1y ago
It’s very odd when mainframes (S/3x0, Cray, yadda yadda) have been extensively water-cooled for over 50 years, and super-dense HPC data centers have used liquid cooling for at least 20, to hear Google-scale data center design compared to PC hobbyist rigs. Selective amnesia + laughably off-target point of comparison.
- jeffbee 1y ago
  Hyper-scale data centers normally need not be concerned with power density, and their designers might avoid density because of the problems it causes. Arguably modern HPC clusters that are still concerned about density are probably misguided. But when it comes to ML workloads, putting everything physically close together starts to bring benefits in terms of interconnectivity.
  - jonathaneunice 1y ago
    LOLWUT? Hyperscalers and HPC data centers have been very concerned about power and thermal density for decades IME. If you're just running web sites or VM farms, sure keep racks cooler and more power efficient. But for those that deeply care about performance, distance drives latency. That drives a huge demand to "pack things closely" and that drives thermal density up, up, up. "Hot enough to roast a pig" was a fintech data center meme of 20 years go, back at 20kW+ racks. Today you're not really cooking until you get north of 30kW.
    - jeffbee 1y ago
      Name a hyper-scale data center where the size and shape suggests that power density ever entered the conversation.
    - michaelt 1y ago
      My impression is a lot of modern data centres are total-power-constrained, but not really space-constrained. At least, if you look at them on streetview a lot of them seem to be in the middle of nowhere, surrounded by miles of undeveloped scrubland. If Google's Henderson, NV data centre [1] needed more space they could simply buy out the adjacent car wrecking yard or pet crematorium, or reconsider the gigantic gatehouse and vast expanses of beige gravel. Even in Belgium, with its higher population density [2] the car park is bigger than the data centre. The layout makes me think they were told they could only have a certain amount of power, but essentially as much land as they needed. So they're concerned about power and thermals, but maybe not about power and thermal density so much. [1] https://www.google.com/maps/place/Google+Data+Center+-+Henderson,+NV/@36.0577493,-115.0102648,663m/ https://www.google.com/maps/place/Google+Data+Center+-+Hende... [2] https://www.google.com/maps/place/Google+Data+Center/@55.5571564,9.649724,758m/ https://www.google.com/maps/place/Google+Data+Center/@55.557...
BoppreH 1y ago
I see frequent mentions of AI wasting water. Is this one such setup, perhaps with the CDU using the facility's water supply for evaporative cooling?
- deleted 1y ago
  [deleted]
- bri3d 1y ago
  The CDU is inside the datacenter and strictly liquid to liquid exchange. It transfers heat from the rack block's coolant to the facility coolant. The facility then provides outdoor heat exchange for the facility coolant, which is sometimes accomplished using open-loop evaporative cooling (spraying down the cooling towers). All datacenters have some form of facility cooling, whether there's a CDU and local water cooling or not, so it's not particularly relevant. The whole AI-water conversation is sort of tiring, since water just moves to more or less efficient parts or locations in the water cycle - I think a "total runtime energy consumption" metric would be much more useful if it were possible to accurately price in water-related externalities (ie - is a massive amount of energy spent moving water because a datacenter evaporates it? or is it no big deal?). And the whole thing really just shows how inefficient and inaccurately priced the market for water is, especially in the US where water rights, price, and the actual utility of water in a given location are often shockingly uncorrelated.
  - maartin0 1y ago
    Slightly old but interesting https://blog.equinix.com/blog/2024/09/19/how-data-centers-use-water-and-how-were-working-to-use-water-responsibly/ https://blog.equinix.com/blog/2024/09/19/how-data-centers-us...
  - deleted 1y ago
    [deleted]
  - bjackman 1y ago
    Even without AI the whole water conservation topic is kinda silly. The margin at which it makes sense to save water varies wildly by location, but the cultural dominance of the western USA infiltrates everything. Here in Zürich, my office briefly installed water saving taps. This is a city of less than half a million where the government maintains 1,200 fountains that spew drinkable water 24/7. But someone at the California HQ said saving water is important and someone at the Swiss subsidiary said yes sir we'll look into it right away sir.
m463 1y ago
I wonder what the economics of water cooling really is. Is it because chips are getting more expensive, so it is more economical to run them faster by liquid cooling them? Or is it data center footprint is more expensive, so denser liquid cooling makes more sense? Or is it that wiring distances (1ft = 1nanosecond) make dense computing faster and more efficient?
- summerlight 1y ago
  Not sure about classical computing demands, but I think wiring distances definitely matter for TPU-like memory heavy computation.
- moffkalast 1y ago
  It's more of a testament to inefficiency, with rising TDP year after year as losses get larger with smaller nm processes. It's so atrocious, even in the consumer sector Nvidia can't even design a connector that doesn't melt during normal usage because their power draw has become beyond absurd. People don't really complain about crappy shovels during a gold rush though unfortunately, they're just happy they got one before they ran out. They have no incentive to innovate in efficiency while the performance line keeps going up.
- MurkyLabs 1y ago
  It's a mixture of both 2 and 3. The chips are getting hotter because they're compacting more stuff in a small space and throwing more power into them. At the same time, powering all those fans that cool the computers takes a lot of power (when you have racks and racks those small fans add up quickly) and that heat is then blown into hot isles that need to then circulate the heat to A/C units. With liquid cooling they're able to save costs due to lower electricity usage and having direct liquid to liquid cool as apposed to chip->air->AC->liquid. ServeTheHome did a write up on it last year, https://www.servethehome.com/estimating-the-power-consumption-impact-of-liquid-cooling-supermicro/ https://www.servethehome.com/estimating-the-power-consumptio...
  - mikepurvis 1y ago
    I've never done DC ops, but I bet fan failure is a factor too— basically there'd be a benefit to centralizing all the cooling for N racks in 2-3 large redundant pumps rather than having each node bringing its own battalion of fans that are all going to individually fail in a bell curve centered on 30k hours of operation, with each failure knocking out the system and requiring hands-on maintenance.
hnburnsy 1y ago
>"Google extensively validates components with leak testing, uses alerting systems to discover problems like leaks, and takes preventative measures like scheduled maintenance and filtration. They also have a clear set of protocols to respond to alerts and issues, allowing a large workforce to respond to problems in a consistent fashion. It’s a far cry from the ad-hoc measures enthusiasts take to maintain their water cooling setups." How about the author compare it to MS, Meta, or AWS instead of Joe Blow buying parts online? I would hope that Google had extensive validation and clear protocols. [Roll-Eyes]
- MBCook 1y ago
  My guess would be the average reader is probably far more familiar with people water cooling their own machines than the intricacies of how various large companies are doing water cooling (or not) in their data centers.
hnburnsy 1y ago
Kind of related from B1M... >The Paris Olympic Pool is Heated by the Internet >https://www.youtube.com/watch?v=2gWudPtN6z4&t=4s https://www.youtube.com/watch?v=2gWudPtN6z4&t=4s
miohtama 1y ago
How much moat Google has in the AI race due to its in-house expertise in data centers and in-house IPR in TPUs? Can any company on Earth match them?
- owebmaster 1y ago
  Not enough to prevent open source models making it all useless, I think.
ykl 1y ago
I once saw an interview with the SVP who oversees Azure datacenter buildout or something like that and a thing that stuck with me was that he said his job got a lot easier when he realized he was no longer in the computer business, he was now in the industrial cooling business. Reading this article immediately made me think back to that.
- kak9 1y ago
  curious to see that interview, do you remember link?
nwmcsween 1y ago
What I don't understand is why not use what I've seen in industrial processes? Why not a low boiling point liquid and a heat exchanger? Or even better power a turbine like biogas plants do
ianpenney 1y ago
Between 2006 and 2012 I was in DCs. I didn’t live there but I had to visit at stupid o’clock too often. It’s not a hospitable place. I would have loved it if cooling was quieter and less extreme. The reason interfaces are on the back is that’s the intake side. So, bring a sweater. Or just go to the other side of the rack to warm your hands back up.
pjdesno 1y ago
In theory data center cooling is simple - CPUs run at 60-70C, while the outside ambient is usually 30C or less, so heat should just "flow downhill" with a bit of help from fans, pumps, etc. The problem with using air cooling to get it there is that the humans who run the data center have to enter and breathe the same air that's used to cool the computers, and if your working fluid gets too hot it's quite unhealthy for them. (we run our hot aisles at 100F, which is a bit toasty, and every third rack is a heat exchanger running off the chilled water lines from the outside evaporative cooler, modulo a heat exchanger to keep the bird shit out) We're not going to be able to pump much heat into the outside world unless our working fluid is a decent amount hotter than ambient, so when it gets reasonably warm outside we need to put chillers (water-to-water AC units) in the loop, which consume energy to basically concentrate that heat into a higher-temperature exhaust. When it's really hot outside they consume quite a bit of energy. If the entire data center was liquid cooled we could have coolant coming from the racks at a much higher temperature, and we'd be able to dump the heat outside on the hottest days without needing any chillers in the loop. As it is we have some liquid cooled racks running off the same chilled water lines as the in-aisle heat exchangers, but the coolant temp is limited by the temperature of our hot aisles, which are quite hot enough already, thank you.
- fanf2 1y ago
  15 years ago IBM installed a supercomputer at ETH Zurich that used 60C hot water as its coolant, with a heat exchanger to the building’s hot water system (which is typically somewhat less than 60C) https://en.m.wikipedia.org/wiki/Aquasar https://en.m.wikipedia.org/wiki/Aquasar
- saagarjha 1y ago
  I wonder if we will eventually have people entering datacenters in heatsuits to eke out some extra efficiency gains from just needing to cool people.
- eternauta3k 1y ago
  > CPUs run at 60-70C, while the outside ambient is usually 30C or less, so heat should just "flow downhill" with a bit of help from fans, pumps, etc. This isn't how you think of heat flow. The CPUs run at a given power. Their temperature will depend on the ambient temperature, and on the thermal impedance between them and the ambient. If the thermal impedance is too high, they'll be too hot and die.
adolph 1y ago
Seeing in "installation & commissioning" mention of "Leak testing" reminds me of the PowerMac G5 liquid cooling leaks: https://www.xlr8yourmac.com/systems/G5_CoolantLeak_Repair/G5_CoolantLeak_Repair.html https://www.xlr8yourmac.com/systems/G5_CoolantLeak_Repair/G5...
tyingq 1y ago
OVH has been doing liquid cooling for quite a while, and is now using immersion also. Somewhat surprising to see at that dirt cheap end of the market.
ehnto 1y ago
I wish I could be as excited as I used to be about Google's infrastructure. But with their constant attacks on the freedom of the internet, they have exhausted my good-will. So I am afraid I do not care about your liquid cooling on this day. I'm also not particularly impressed, I am sure the devil is in the details at this scale but there doesn't seem to be much novel here. If a Googler happens to read this and it makes them sad, it's Google, not you. You might consider doing cool stuff somewhere else.
- anon6362 1y ago
  FAANGs have gone from wunderkind darlings that would pave the way for progress, like turn of the 20th-century-style, to monopolistic, feudal overlords rapaciously seeking to exploit land and resources with zero concern for the environment or locals' quality of life.
flakes 1y ago
> In theory, a PC builder could try exchanging heat to a toilet tank, allowing for highly efficient cooling with each flush. The future is here.
- WJW 1y ago
  In an interesting way, the reverse is also here: some district heating plants use heat pumps to extract waste heat from sewage to provide efficient heat for district heating systems. See https://www.eneco.nl/en/about-us/what-we-do/sustainable-resources/heat/heat-pump-at-utrecht-sewage-treatment-plant/how-does-the-heat-pump-work/ https://www.eneco.nl/en/about-us/what-we-do/sustainable-reso... for a basic overview.
  - lopis 1y ago
    I dream of a future where there's an efficient energy recapture in the house. We waste a lot of heat in the shower, and washing clothes and dishes. We also run the fridge all winter when it's less than 5C outside, and the fridge then produces a lot of heat. And when we're done cooking, the oven is still really hot. Could the home heating system of the future use all this heat (and cold) more efficiently?
    - mosquitobiten 1y ago
      I don't see this happening until we have ordinary people living in orbit.
    - mnw21cam 1y ago
      The heat in the oven will slowly leak out into the rest of the house, so the home heating system of the right now already uses that. The fridge example is much more relevant though. If you could get the hot-side radiator part of the fridge thermally-connected with the outside temperature, then it would run much more efficiently. However, you'd lose that heat from the house, so if you're having to heat the house as well it might not be worth it.
      - 1y ago
bilater 1y ago
I’ve always wondered: what if we could actually use this heat? For example, could we take the heated water from cooling a data center and pump it to places that need it (Greenhouses, Industrial processes, Desalination and water treatment)?
- xxpor 1y ago
  Happens for urban data centers/internet exchanges: www.aboutamazon.com/news/sustainability/the-super-efficient-heat-source-hidden-below-amazons-seattle-headquarters
- rockostrich 1y ago
  It's done sparingly. If you search for "data center waste heat uses" you'll find cases like you mention [1] [2] but the ROI on implementing it seems to be lower than renewable energy infrastructure. [1] https://www.informationweek.com/sustainability/reusing-waste-heat-from-data-centers-to-make-things-grow https://www.informationweek.com/sustainability/reusing-waste... [2] https://www.techspot.com/news/97995-data-center-uses-waste-heat-warm-public-pool.html https://www.techspot.com/news/97995-data-center-uses-waste-h...
anon6362 1y ago
That's pretty much the way it had to go, other than doing full immersion cooling like GRC that never caught on. Meta has various rack-based chill water heat-exchanger cooling solutions but this looks a bit more integrated.

Relevant. OVH has been using liquid cooling for many years: https://blog.ovhcloud.com/new-hybrid-immersion-liquid-cooling-developments-at-ovhcloud/ https://blog.ovhcloud.com/new-hybrid-immersion-liquid-coolin... https://www.youtube.com/watch?v=S_DHJcQpumI https://www.youtube.com/watch?v=S_DHJcQpumI

jeffbee 1y ago
And yet their claimed PUE is 1.26 which is pretty bad. One way to characterize that overall PUE figure is they waste 3x as much on overhead as Google (claimed 1.09 global PUE) or Meta (1.08).
- Loic 1y ago
  10 years ago, OVH datacenter SBG1 was below 1.1 with a mixture of natural convection + liquid cooling. But I have not been following in the latest years.
bri3d 1y ago
Your linked articles are about immersion cooling, which is "liquid cooling," I suppose, but a very different thing. Do OVH actually use immersion cooling in production? This seems like a "labs" product that hasn't been fully baked yet. OVH _do_ definitely use traditional water-loop/water-block liquid cooling like Google are presenting, described here: https://blog.ovhcloud.com/understanding-ovhclouds-data-centres-how-delta-t-helps-with-water-efficiency/ https://blog.ovhcloud.com/understanding-ovhclouds-data-centr... and visually in https://www.youtube.com/watch?v=RFzirpvTiOo https://www.youtube.com/watch?v=RFzirpvTiOo , but while it looks very similar architecturally their setup seems almost comically inefficient compared to Google's according to their PUE disclosures.
- TiredOfLife 1y ago
  Based on how their data centers burn down, they don't even use liquid in their fire suppression systems.

michaelt 1y ago

> TPU chips are hooked up in series in the loop, which naturally means some chips will get hotter liquid that has already passed other chips in the loop. Cooling capacity is budgeted based on the requirements of the last chip in each loop. Of course, it's worth noting that if you've got four chips, each putting out 250W of power, and a pump pushing 1 litres of water per minute through them, water at the outlet must be 14°C hotter than water at the inlet, because of the specific heat capacity of water. That's true whether the water flows through the chips in series, or in parallel.

foota 1y ago
Hm... but in the case when the chips are in serial, the heat transfer from the last chip will be less than when the chips are in parallel, because the rate of heating is proportional to the difference in temperature, and the water starts at a lower temperature for the parallel case for this last theoretical chip.
- fraserphysics 1y ago
  One way to characterize the cost of cooling is entropy production. As you say, cooling is proportional to difference in temperature. However, entropy production is also proportional to temperature difference. It's not my field, but it looks like an interesting challenge to optimize competing objectives.
- 0x457 1y ago
  Yes, but water is constantly moving in a loop. It's not like you use water to cool chip #1, and then it moves to chip #2, it's constantly moving, so temperature delta isn't that much.
  - smachiz 1y ago
    in their first serial design, that's exactly what it was doing.
    - Dylan16807 1y ago

jonathaneunice 1y ago

It’s very odd when mainframes (S/3x0, Cray, yadda yadda) have been extensively water-cooled for over 50 years, and super-dense HPC data centers have used liquid cooling for at least 20, to hear Google-scale data center design compared to PC hobbyist rigs. Selective amnesia + laughably off-target point of comparison.

jeffbee 1y ago
Hyper-scale data centers normally need not be concerned with power density, and their designers might avoid density because of the problems it causes. Arguably modern HPC clusters that are still concerned about density are probably misguided. But when it comes to ML workloads, putting everything physically close together starts to bring benefits in terms of interconnectivity.
- jonathaneunice 1y ago
  LOLWUT? Hyperscalers and HPC data centers have been very concerned about power and thermal density for decades IME. If you're just running web sites or VM farms, sure keep racks cooler and more power efficient. But for those that deeply care about performance, distance drives latency. That drives a huge demand to "pack things closely" and that drives thermal density up, up, up. "Hot enough to roast a pig" was a fintech data center meme of 20 years go, back at 20kW+ racks. Today you're not really cooking until you get north of 30kW.
  - jeffbee 1y ago
    Name a hyper-scale data center where the size and shape suggests that power density ever entered the conversation.
  - michaelt 1y ago
    My impression is a lot of modern data centres are total-power-constrained, but not really space-constrained. At least, if you look at them on streetview a lot of them seem to be in the middle of nowhere, surrounded by miles of undeveloped scrubland. If Google's Henderson, NV data centre [1] needed more space they could simply buy out the adjacent car wrecking yard or pet crematorium, or reconsider the gigantic gatehouse and vast expanses of beige gravel. Even in Belgium, with its higher population density [2] the car park is bigger than the data centre. The layout makes me think they were told they could only have a certain amount of power, but essentially as much land as they needed. So they're concerned about power and thermals, but maybe not about power and thermal density so much. [1] https://www.google.com/maps/place/Google+Data+Center+-+Henderson,+NV/@36.0577493,-115.0102648,663m/ https://www.google.com/maps/place/Google+Data+Center+-+Hende... [2] https://www.google.com/maps/place/Google+Data+Center/@55.5571564,9.649724,758m/ https://www.google.com/maps/place/Google+Data+Center/@55.557...

BoppreH 1y ago

I see frequent mentions of AI wasting water. Is this one such setup, perhaps with the CDU using the facility's water supply for evaporative cooling?

deleted 1y ago
[deleted]
bri3d 1y ago
The CDU is inside the datacenter and strictly liquid to liquid exchange. It transfers heat from the rack block's coolant to the facility coolant. The facility then provides outdoor heat exchange for the facility coolant, which is sometimes accomplished using open-loop evaporative cooling (spraying down the cooling towers). All datacenters have some form of facility cooling, whether there's a CDU and local water cooling or not, so it's not particularly relevant. The whole AI-water conversation is sort of tiring, since water just moves to more or less efficient parts or locations in the water cycle - I think a "total runtime energy consumption" metric would be much more useful if it were possible to accurately price in water-related externalities (ie - is a massive amount of energy spent moving water because a datacenter evaporates it? or is it no big deal?). And the whole thing really just shows how inefficient and inaccurately priced the market for water is, especially in the US where water rights, price, and the actual utility of water in a given location are often shockingly uncorrelated.
- maartin0 1y ago
  Slightly old but interesting https://blog.equinix.com/blog/2024/09/19/how-data-centers-use-water-and-how-were-working-to-use-water-responsibly/ https://blog.equinix.com/blog/2024/09/19/how-data-centers-us...
- deleted 1y ago
  [deleted]
- bjackman 1y ago
  Even without AI the whole water conservation topic is kinda silly. The margin at which it makes sense to save water varies wildly by location, but the cultural dominance of the western USA infiltrates everything. Here in Zürich, my office briefly installed water saving taps. This is a city of less than half a million where the government maintains 1,200 fountains that spew drinkable water 24/7. But someone at the California HQ said saving water is important and someone at the Swiss subsidiary said yes sir we'll look into it right away sir.

m463 1y ago

I wonder what the economics of water cooling really is. Is it because chips are getting more expensive, so it is more economical to run them faster by liquid cooling them? Or is it data center footprint is more expensive, so denser liquid cooling makes more sense? Or is it that wiring distances (1ft = 1nanosecond) make dense computing faster and more efficient?

summerlight 1y ago
Not sure about classical computing demands, but I think wiring distances definitely matter for TPU-like memory heavy computation.
moffkalast 1y ago
It's more of a testament to inefficiency, with rising TDP year after year as losses get larger with smaller nm processes. It's so atrocious, even in the consumer sector Nvidia can't even design a connector that doesn't melt during normal usage because their power draw has become beyond absurd. People don't really complain about crappy shovels during a gold rush though unfortunately, they're just happy they got one before they ran out. They have no incentive to innovate in efficiency while the performance line keeps going up.
MurkyLabs 1y ago
It's a mixture of both 2 and 3. The chips are getting hotter because they're compacting more stuff in a small space and throwing more power into them. At the same time, powering all those fans that cool the computers takes a lot of power (when you have racks and racks those small fans add up quickly) and that heat is then blown into hot isles that need to then circulate the heat to A/C units. With liquid cooling they're able to save costs due to lower electricity usage and having direct liquid to liquid cool as apposed to chip->air->AC->liquid. ServeTheHome did a write up on it last year, https://www.servethehome.com/estimating-the-power-consumption-impact-of-liquid-cooling-supermicro/ https://www.servethehome.com/estimating-the-power-consumptio...
- mikepurvis 1y ago
  I've never done DC ops, but I bet fan failure is a factor too— basically there'd be a benefit to centralizing all the cooling for N racks in 2-3 large redundant pumps rather than having each node bringing its own battalion of fans that are all going to individually fail in a bell curve centered on 30k hours of operation, with each failure knocking out the system and requiring hands-on maintenance.

hnburnsy 1y ago

>"Google extensively validates components with leak testing, uses alerting systems to discover problems like leaks, and takes preventative measures like scheduled maintenance and filtration. They also have a clear set of protocols to respond to alerts and issues, allowing a large workforce to respond to problems in a consistent fashion. It’s a far cry from the ad-hoc measures enthusiasts take to maintain their water cooling setups." How about the author compare it to MS, Meta, or AWS instead of Joe Blow buying parts online? I would hope that Google had extensive validation and clear protocols. [Roll-Eyes]

MBCook 1y ago
My guess would be the average reader is probably far more familiar with people water cooling their own machines than the intricacies of how various large companies are doing water cooling (or not) in their data centers.

hnburnsy 1y ago

Kind of related from B1M... >The Paris Olympic Pool is Heated by the Internet >https://www.youtube.com/watch?v=2gWudPtN6z4&t=4s https://www.youtube.com/watch?v=2gWudPtN6z4&t=4s

miohtama 1y ago

How much moat Google has in the AI race due to its in-house expertise in data centers and in-house IPR in TPUs? Can any company on Earth match them?

owebmaster 1y ago
Not enough to prevent open source models making it all useless, I think.

ykl 1y ago

I once saw an interview with the SVP who oversees Azure datacenter buildout or something like that and a thing that stuck with me was that he said his job got a lot easier when he realized he was no longer in the computer business, he was now in the industrial cooling business. Reading this article immediately made me think back to that.

kak9 1y ago
curious to see that interview, do you remember link?

nwmcsween 1y ago

What I don't understand is why not use what I've seen in industrial processes? Why not a low boiling point liquid and a heat exchanger? Or even better power a turbine like biogas plants do

ianpenney 1y ago

Between 2006 and 2012 I was in DCs. I didn’t live there but I had to visit at stupid o’clock too often. It’s not a hospitable place. I would have loved it if cooling was quieter and less extreme. The reason interfaces are on the back is that’s the intake side. So, bring a sweater. Or just go to the other side of the rack to warm your hands back up.

pjdesno 1y ago

In theory data center cooling is simple - CPUs run at 60-70C, while the outside ambient is usually 30C or less, so heat should just "flow downhill" with a bit of help from fans, pumps, etc. The problem with using air cooling to get it there is that the humans who run the data center have to enter and breathe the same air that's used to cool the computers, and if your working fluid gets too hot it's quite unhealthy for them. (we run our hot aisles at 100F, which is a bit toasty, and every third rack is a heat exchanger running off the chilled water lines from the outside evaporative cooler, modulo a heat exchanger to keep the bird shit out) We're not going to be able to pump much heat into the outside world unless our working fluid is a decent amount hotter than ambient, so when it gets reasonably warm outside we need to put chillers (water-to-water AC units) in the loop, which consume energy to basically concentrate that heat into a higher-temperature exhaust. When it's really hot outside they consume quite a bit of energy. If the entire data center was liquid cooled we could have coolant coming from the racks at a much higher temperature, and we'd be able to dump the heat outside on the hottest days without needing any chillers in the loop. As it is we have some liquid cooled racks running off the same chilled water lines as the in-aisle heat exchangers, but the coolant temp is limited by the temperature of our hot aisles, which are quite hot enough already, thank you.

fanf2 1y ago
15 years ago IBM installed a supercomputer at ETH Zurich that used 60C hot water as its coolant, with a heat exchanger to the building’s hot water system (which is typically somewhat less than 60C) https://en.m.wikipedia.org/wiki/Aquasar https://en.m.wikipedia.org/wiki/Aquasar
saagarjha 1y ago
I wonder if we will eventually have people entering datacenters in heatsuits to eke out some extra efficiency gains from just needing to cool people.
eternauta3k 1y ago
> CPUs run at 60-70C, while the outside ambient is usually 30C or less, so heat should just "flow downhill" with a bit of help from fans, pumps, etc. This isn't how you think of heat flow. The CPUs run at a given power. Their temperature will depend on the ambient temperature, and on the thermal impedance between them and the ambient. If the thermal impedance is too high, they'll be too hot and die.

adolph 1y ago

Seeing in "installation & commissioning" mention of "Leak testing" reminds me of the PowerMac G5 liquid cooling leaks: https://www.xlr8yourmac.com/systems/G5_CoolantLeak_Repair/G5_CoolantLeak_Repair.html https://www.xlr8yourmac.com/systems/G5_CoolantLeak_Repair/G5...

tyingq 1y ago

OVH has been doing liquid cooling for quite a while, and is now using immersion also. Somewhat surprising to see at that dirt cheap end of the market.

ehnto 1y ago

I wish I could be as excited as I used to be about Google's infrastructure. But with their constant attacks on the freedom of the internet, they have exhausted my good-will. So I am afraid I do not care about your liquid cooling on this day. I'm also not particularly impressed, I am sure the devil is in the details at this scale but there doesn't seem to be much novel here. If a Googler happens to read this and it makes them sad, it's Google, not you. You might consider doing cool stuff somewhere else.

anon6362 1y ago
FAANGs have gone from wunderkind darlings that would pave the way for progress, like turn of the 20th-century-style, to monopolistic, feudal overlords rapaciously seeking to exploit land and resources with zero concern for the environment or locals' quality of life.

flakes 1y ago

> In theory, a PC builder could try exchanging heat to a toilet tank, allowing for highly efficient cooling with each flush. The future is here.

WJW 1y ago
In an interesting way, the reverse is also here: some district heating plants use heat pumps to extract waste heat from sewage to provide efficient heat for district heating systems. See https://www.eneco.nl/en/about-us/what-we-do/sustainable-resources/heat/heat-pump-at-utrecht-sewage-treatment-plant/how-does-the-heat-pump-work/ https://www.eneco.nl/en/about-us/what-we-do/sustainable-reso... for a basic overview.
- lopis 1y ago
  I dream of a future where there's an efficient energy recapture in the house. We waste a lot of heat in the shower, and washing clothes and dishes. We also run the fridge all winter when it's less than 5C outside, and the fridge then produces a lot of heat. And when we're done cooking, the oven is still really hot. Could the home heating system of the future use all this heat (and cold) more efficiently?
  - mosquitobiten 1y ago
    I don't see this happening until we have ordinary people living in orbit.
  - mnw21cam 1y ago
    The heat in the oven will slowly leak out into the rest of the house, so the home heating system of the right now already uses that. The fridge example is much more relevant though. If you could get the hot-side radiator part of the fridge thermally-connected with the outside temperature, then it would run much more efficiently. However, you'd lose that heat from the house, so if you're having to heat the house as well it might not be worth it.
    - 1y ago

bilater 1y ago

I’ve always wondered: what if we could actually use this heat? For example, could we take the heated water from cooling a data center and pump it to places that need it (Greenhouses, Industrial processes, Desalination and water treatment)?

xxpor 1y ago
Happens for urban data centers/internet exchanges: www.aboutamazon.com/news/sustainability/the-super-efficient-heat-source-hidden-below-amazons-seattle-headquarters
rockostrich 1y ago
It's done sparingly. If you search for "data center waste heat uses" you'll find cases like you mention [1] [2] but the ROI on implementing it seems to be lower than renewable energy infrastructure. [1] https://www.informationweek.com/sustainability/reusing-waste-heat-from-data-centers-to-make-things-grow https://www.informationweek.com/sustainability/reusing-waste... [2] https://www.techspot.com/news/97995-data-center-uses-waste-heat-warm-public-pool.html https://www.techspot.com/news/97995-data-center-uses-waste-h...

anon6362 1y ago

That's pretty much the way it had to go, other than doing full immersion cooling like GRC that never caught on. Meta has various rack-based chill water heat-exchanger cooling solutions but this looks a bit more integrated.