6 ms·

It’s very odd when mainframes (S/3x0, Cray, yadda yadda) have been extensively water-cooled for over 50 years, and super-dense HPC data centers have used liquid

It’s very odd when mainframes (S/3x0, Cray, yadda yadda) have been extensively water-cooled for over 50 years, and super-dense HPC data centers have used liquid cooling for at least 20, to hear Google-scale data center design compared to PC hobbyist rigs. Selective amnesia + laughably off-target point of comparison.

jeffbee 1y ago
Hyper-scale data centers normally need not be concerned with power density, and their designers might avoid density because of the problems it causes. Arguably modern HPC clusters that are still concerned about density are probably misguided. But when it comes to ML workloads, putting everything physically close together starts to bring benefits in terms of interconnectivity.
- jonathaneunice 1y ago
  LOLWUT? Hyperscalers and HPC data centers have been very concerned about power and thermal density for decades IME. If you're just running web sites or VM farms, sure keep racks cooler and more power efficient. But for those that deeply care about performance, distance drives latency. That drives a huge demand to "pack things closely" and that drives thermal density up, up, up. "Hot enough to roast a pig" was a fintech data center meme of 20 years go, back at 20kW+ racks. Today you're not really cooking until you get north of 30kW.
  - jeffbee 1y ago
    Name a hyper-scale data center where the size and shape suggests that power density ever entered the conversation.
  - michaelt 1y ago
    My impression is a lot of modern data centres are total-power-constrained, but not really space-constrained. At least, if you look at them on streetview a lot of them seem to be in the middle of nowhere, surrounded by miles of undeveloped scrubland. If Google's Henderson, NV data centre [1] needed more space they could simply buy out the adjacent car wrecking yard or pet crematorium, or reconsider the gigantic gatehouse and vast expanses of beige gravel. Even in Belgium, with its higher population density [2] the car park is bigger than the data centre. The layout makes me think they were told they could only have a certain amount of power, but essentially as much land as they needed. So they're concerned about power and thermals, but maybe not about power and thermal density so much. [1] https://www.google.com/maps/place/Google+Data+Center+-+Henderson,+NV/@36.0577493,-115.0102648,663m/ https://www.google.com/maps/place/Google+Data+Center+-+Hende... [2] https://www.google.com/maps/place/Google+Data+Center/@55.5571564,9.649724,758m/ https://www.google.com/maps/place/Google+Data+Center/@55.557...
spankalee 1y ago
From the article: > Liquid cooling is a familiar concept to PC enthusiasts, and has a long history in enterprise compute as well. And the trend in data centers was to move towards more passive cooling at the individual servers and hotter operating temperatures for a while. This is interesting because it reverses that trend a lot, and possibly because of the per-row cooling.
- echelon 1y ago
  This abundance of marketing (not necessarily this blog post) is happening because of all the environmental chatter about AI and data centers recently. Google wants you to know it recycles its water. It's free points. Edit: to clarify, normal social media is being flooded with stories about AI energy and water usage. Google isn't greenwashing, they're simply showing how things work and getting good press for something they already do.
  - mlyle 1y ago
    Google uses plenty of water in cooling towers and chillers. Sure, water cooling loops to the server may reduce the amount a little bit compared to fans, but this isn't "recycling" in any normal sense. It's mostly a play for density.
  - Legend2440 1y ago
    The environmental impact of water usage seems way overblown to me. Last year, U.S. data centers consumed 17 billion gallons of water. Which sounds like a lot, but the US as a whole uses 300 billion gallons of water every day. Water is not a scarce resource in much of the country.
    - foota 1y ago
      Personally, I agree. That said, I think it might be worth considering the impact of water usage in local areas that are relatively water constrained.
    - Guvante 1y ago
legulere 1y ago
It's not so surprising when considering googles history coming from inexpensive commodity hardware. It's pretty similar to how it took decades for x86 servers and operating systems to gain mainframe functionality like virtualisation. https://blog.codinghorror.com/building-a-computer-the-google-way/ https://blog.codinghorror.com/building-a-computer-the-google...
- mr_toad 1y ago
  Once again IT comes full circle.
liquidgecka 1y ago
[bri3d pointed out that I missed an element of this. There is a transfer between rack level and machine level coolant which makes this far less novel than I had initially understood. See their direct comment to this] I posed this further down in a reply-to-a-reply but I should call it out a little closer to the top: The innovation here is not “we are using water for cooling”. The innovation here is that they are direct cooling the servers with chillers that are outside of the facility. Most mainframes will use water cooling to get the heat from the core out to the edges where traditional where it can be picked up by the typical heatsink/cooling fans. Even home PCs do this by moving the heat to a reservoir that can be more effectively cooled. What Google is doing is using the huge chillers that would normally be cooling the air in the facility to cool water which is directly pumped into every server. The return water is then cooled in the chiller tower. This eliminates ANY air based transfer besides the chiller tower. This is one being done a server or a rack.. its being done on the whole data center all at once. I am super curious how they handle things like chiller maintenance or pump failures. I am sure they have redundancy but the system for that has to be super impressive because it can’t be offline long before you experience hardware failure! [Edit: It was pointed out in another comment that AWS is doing this as well and honestly their pictures make it way clearer what is happening: https://www.aboutamazon.com/news/aws/aws-liquid-cooling-data-centers https://www.aboutamazon.com/news/aws/aws-liquid-cooling-data...]
- ambicapter 1y ago
  So every time they plug in a server they also plug in water lines?
  - jedberg 1y ago
    Looks like it. New server means power, internet, and water.
    - fudgy73 1y ago
      just like humans.
  - 1y ago
coliveira 1y ago
[flagged]
- jsnell 1y ago
  Why make up such an absurd grievance to get fake-outraged about? Nowhere in the article does it say that Google claims to have invented liquid cooling. In fact, nowhere does it say they claim any part of this is a new invention. But the point of this kind of paper is typically not what is new, it's what combination of previously known and novel techniques have been found to work well at massive scale over a timespan of years.
bluedino 1y ago
> super-dense HPC data centers have used liquid cooling for at least 20 Hasn't this just been for things like rack doors and such? In the last ~two generations of servers it seems like now there's finally DLC (direct liquid cooling) into the actual servers themselves (similar to the article). Intel kind of forced that one on us, with their high-end SKUs. This has been a pain becuase it doesn't fit into legacy datacenters as easily as the door/rack-based systems. I won't say which server vendor it is, but I've put in more than one service ticket for leaking coolant bags (they hang them on the racks).

jeffbee 1y ago

Hyper-scale data centers normally need not be concerned with power density, and their designers might avoid density because of the problems it causes. Arguably modern HPC clusters that are still concerned about density are probably misguided. But when it comes to ML workloads, putting everything physically close together starts to bring benefits in terms of interconnectivity.

jonathaneunice 1y ago
LOLWUT? Hyperscalers and HPC data centers have been very concerned about power and thermal density for decades IME. If you're just running web sites or VM farms, sure keep racks cooler and more power efficient. But for those that deeply care about performance, distance drives latency. That drives a huge demand to "pack things closely" and that drives thermal density up, up, up. "Hot enough to roast a pig" was a fintech data center meme of 20 years go, back at 20kW+ racks. Today you're not really cooking until you get north of 30kW.
- jeffbee 1y ago
  Name a hyper-scale data center where the size and shape suggests that power density ever entered the conversation.
- michaelt 1y ago
  My impression is a lot of modern data centres are total-power-constrained, but not really space-constrained. At least, if you look at them on streetview a lot of them seem to be in the middle of nowhere, surrounded by miles of undeveloped scrubland. If Google's Henderson, NV data centre [1] needed more space they could simply buy out the adjacent car wrecking yard or pet crematorium, or reconsider the gigantic gatehouse and vast expanses of beige gravel. Even in Belgium, with its higher population density [2] the car park is bigger than the data centre. The layout makes me think they were told they could only have a certain amount of power, but essentially as much land as they needed. So they're concerned about power and thermals, but maybe not about power and thermal density so much. [1] https://www.google.com/maps/place/Google+Data+Center+-+Henderson,+NV/@36.0577493,-115.0102648,663m/ https://www.google.com/maps/place/Google+Data+Center+-+Hende... [2] https://www.google.com/maps/place/Google+Data+Center/@55.5571564,9.649724,758m/ https://www.google.com/maps/place/Google+Data+Center/@55.557...

spankalee 1y ago

From the article: > Liquid cooling is a familiar concept to PC enthusiasts, and has a long history in enterprise compute as well. And the trend in data centers was to move towards more passive cooling at the individual servers and hotter operating temperatures for a while. This is interesting because it reverses that trend a lot, and possibly because of the per-row cooling.

echelon 1y ago
This abundance of marketing (not necessarily this blog post) is happening because of all the environmental chatter about AI and data centers recently. Google wants you to know it recycles its water. It's free points. Edit: to clarify, normal social media is being flooded with stories about AI energy and water usage. Google isn't greenwashing, they're simply showing how things work and getting good press for something they already do.
- mlyle 1y ago
  Google uses plenty of water in cooling towers and chillers. Sure, water cooling loops to the server may reduce the amount a little bit compared to fans, but this isn't "recycling" in any normal sense. It's mostly a play for density.
- Legend2440 1y ago
  The environmental impact of water usage seems way overblown to me. Last year, U.S. data centers consumed 17 billion gallons of water. Which sounds like a lot, but the US as a whole uses 300 billion gallons of water every day. Water is not a scarce resource in much of the country.
  - foota 1y ago
    Personally, I agree. That said, I think it might be worth considering the impact of water usage in local areas that are relatively water constrained.
  - Guvante 1y ago

legulere 1y ago

It's not so surprising when considering googles history coming from inexpensive commodity hardware. It's pretty similar to how it took decades for x86 servers and operating systems to gain mainframe functionality like virtualisation. https://blog.codinghorror.com/building-a-computer-the-google-way/ https://blog.codinghorror.com/building-a-computer-the-google...

mr_toad 1y ago
Once again IT comes full circle.

liquidgecka 1y ago

[bri3d pointed out that I missed an element of this. There is a transfer between rack level and machine level coolant which makes this far less novel than I had initially understood. See their direct comment to this] I posed this further down in a reply-to-a-reply but I should call it out a little closer to the top: The innovation here is not “we are using water for cooling”. The innovation here is that they are direct cooling the servers with chillers that are outside of the facility. Most mainframes will use water cooling to get the heat from the core out to the edges where traditional where it can be picked up by the typical heatsink/cooling fans. Even home PCs do this by moving the heat to a reservoir that can be more effectively cooled. What Google is doing is using the huge chillers that would normally be cooling the air in the facility to cool water which is directly pumped into every server. The return water is then cooled in the chiller tower. This eliminates ANY air based transfer besides the chiller tower. This is one being done a server or a rack.. its being done on the whole data center all at once. I am super curious how they handle things like chiller maintenance or pump failures. I am sure they have redundancy but the system for that has to be super impressive because it can’t be offline long before you experience hardware failure! [Edit: It was pointed out in another comment that AWS is doing this as well and honestly their pictures make it way clearer what is happening: https://www.aboutamazon.com/news/aws/aws-liquid-cooling-data-centers https://www.aboutamazon.com/news/aws/aws-liquid-cooling-data...]

ambicapter 1y ago
So every time they plug in a server they also plug in water lines?
- jedberg 1y ago
  Looks like it. New server means power, internet, and water.
  - fudgy73 1y ago
    just like humans.
- 1y ago

coliveira 1y ago

[flagged]

jsnell 1y ago
Why make up such an absurd grievance to get fake-outraged about? Nowhere in the article does it say that Google claims to have invented liquid cooling. In fact, nowhere does it say they claim any part of this is a new invention. But the point of this kind of paper is typically not what is new, it's what combination of previously known and novel techniques have been found to work well at massive scale over a timespan of years.

bluedino 1y ago

> super-dense HPC data centers have used liquid cooling for at least 20 Hasn't this just been for things like rack doors and such? In the last ~two generations of servers it seems like now there's finally DLC (direct liquid cooling) into the actual servers themselves (similar to the article). Intel kind of forced that one on us, with their high-end SKUs. This has been a pain becuase it doesn't fit into legacy datacenters as easily as the door/rack-based systems. I won't say which server vendor it is, but I've put in more than one service ticket for leaking coolant bags (they hang them on the racks).