8 ms·

[bri3d pointed out that I missed an element of this. There is a transfer between rack level and machine level coolant which makes this far less novel than I had

[bri3d pointed out that I missed an element of this. There is a transfer between rack level and machine level coolant which makes this far less novel than I had initially understood. See their direct comment to this] I posed this further down in a reply-to-a-reply but I should call it out a little closer to the top: The innovation here is not “we are using water for cooling”. The innovation here is that they are direct cooling the servers with chillers that are outside of the facility. Most mainframes will use water cooling to get the heat from the core out to the edges where traditional where it can be picked up by the typical heatsink/cooling fans. Even home PCs do this by moving the heat to a reservoir that can be more effectively cooled. What Google is doing is using the huge chillers that would normally be cooling the air in the facility to cool water which is directly pumped into every server. The return water is then cooled in the chiller tower. This eliminates ANY air based transfer besides the chiller tower. This is one being done a server or a rack.. its being done on the whole data center all at once. I am super curious how they handle things like chiller maintenance or pump failures. I am sure they have redundancy but the system for that has to be super impressive because it can’t be offline long before you experience hardware failure! [Edit: It was pointed out in another comment that AWS is doing this as well and honestly their pictures make it way clearer what is happening: https://www.aboutamazon.com/news/aws/aws-liquid-cooling-data-centers https://www.aboutamazon.com/news/aws/aws-liquid-cooling-data...]

ambicapter 1y ago
So every time they plug in a server they also plug in water lines?
- jedberg 1y ago
  Looks like it. New server means power, internet, and water.
  - fudgy73 1y ago
    just like humans.
- liquidgecka 1y ago
  [I am not a current Google Employee so my understanding of this is based on externally written articles and “leap of faith” guestimation] Yes. A supply and return line along with power. Though if I had to guess how its setup this would be done with some super slick “it just works” kind of mount that lets them just slide the case in and lock it in place. When I was there almost all hardware replacement was made downright trivial so it could just be more or less slide in place and walk away.
  - nielsbot 1y ago
    Maybe similar to a gasoline hose breakaway https://www.opwglobal.com/products/us/retail-fueling-products/standard-dispensing-equipment/swivels-and-breakaways/reconnectable-breakaways https://www.opwglobal.com/products/us/retail-fueling-product...
  - michaelt 1y ago
    Interestingly, entire supercomputers have been decommissioned [1] due to faulty quick disconnects causing water spray. So you can get a single, blind-mating connector combining power, data and water - but you might not want to :) [1] https://gsaauctions.gov/auctions/preview/282996 https://gsaauctions.gov/auctions/preview/282996
  - semi-extrinsic 1y ago
    https://amphenol-industrial.com/products/liquid-cooling-systems/ https://amphenol-industrial.com/products/liquid-cooling-syst...
    - tesseract 1y ago
      CPC is in this market too https://www.cpcworldwide.com/Liquid-Cooling/Products/Blind-Mate https://www.cpcworldwide.com/Liquid-Cooling/Products/Blind-M... Non-spill fluid quick disconnects are established tech in industries like medical, chemical processing, beverage dispensing, and hydraulic power, so there are plenty of design concepts to draw on.
  - scrlk 1y ago
- ajb 1y ago
  I remember reading somewhere that they don't operate at the level of servers; if one dies they leave it in place until they're ready to replace the whole rack. Don't know if that's true now, though. It does sound like connections do involve water lines though. As they are isolating different water circuits, in theory they could have a dry connection between heat exchanger plates, or one made through thermal paste. It doesn't sound like they're doing that though.
  - cavisne 1y ago
    Definitely not true for these workloads. TPUs are interconnected, one dying makes the whole cluster significantly less useful.
  - liquidgecka 1y ago
    It has not been true for a LONG time. That was part of Google early “compute unit” strategy that involved things like sealed containers and such. Turns out that’s not super efficient or useful because you leave large swaths of hardware idle. In my day we had software that would “drain” a machine and release it to hardware ops to swap the hardware on. This could be a drive, memory, CPU or a motherboard. If it was even slightly complicated they would ship it to Mountain View for diagnostic and repair. But every machine was expected to be cycled to get it working as fast as possible. We did a disk upgrade on a whole datacenter that involved switching from 1TB to 2TB disks or something like that (I am dating myself) and total downtime was so important they hired temporary workers to work nights to get the swap done as quickly as possible. If I remember correctly that was part of the “holy cow gmail is out of space!” chaos though, so added urgency.
    - throwaway2037 1y ago
      > part of the “holy cow gmail is out of space!” chaos This sounds like an interesting story. Can you share more details.
    - Cthulhu_ 1y ago
      I'd love to work in a datacenter at that scale sometime, it sounds like it's like working in a warehouse where you get a list of orders, servers to remove and pick up, but at the scales of the Googles et al, that's hundreds of server replacements a day, and production lines of new servers being built and existing ones being repaired or decommissioned. It's a fascinating industry, but only in my head as the only info you get about it is carefully polished articles and the occasional anecdote on HN, which is also carefully polished due to NDAs.
- jayd16 1y ago
  Maybe we can declutter things if they get PWoE(power and water over ethernet) or just a USB-W standard.
  - Nition 1y ago
    It worked for MONIAC.
  - taneq 1y ago
    Great, now I’ll have to figure out if my USB Hi-Flow cable is slowing down my USB Full-Flow drink bottle refilling.
    - toast0 1y ago
      It's not. Full flow is less than hi-flow... Your bottle might slow down other fills on the bus though :p
  - Cthulhu_ 1y ago
    Related, I vaguely recall a concept from a Tesla charger for semis that proposed a charging cable with active coolant flowing through it as well to keep the wire cooled. edit: https://www.teslarati.com/tesla-liquid-cooled-supercharger-cables/ https://www.teslarati.com/tesla-liquid-cooled-supercharger-c...
    - jayd16 1y ago
      This is why I risk jokes on HN. Normally they're frowned upon but it's worth it to find interesting tech that meets the eccentric idea.
- Hilift 1y ago
  And a 12V battery.
nitwit005 1y ago
This was before I was born, so I'm hardly an expert, but I've heard of feeding IBM mainframes chilled water. A quick check of wikipedia found some mention of the idea: https://en.wikipedia.org/wiki/IBM_3090 https://en.wikipedia.org/wiki/IBM_3090
- ChuckMcM 1y ago
  When our mainframe in 1978 sprung a leak in its water cooling jacket it took down the main east/west node on IBMs internal network at the time. :-). But that was definitely a different chilling mechanism than the types Google uses.
- jauntywundrkind 1y ago
  Having to pre chill water (via a refrigeration cycle) is radically less efficient than being able to collect and then disperse heat. It generates considerably more heat ahead of time, to deliver the chilled water. This mode of gathering the heat and sending it out, dealing with the heat after it is produced rather than in advance, should be much more energy efficient. I don't know what surprises me about it so much, but having these rack-sized CDU heat-exchangers was quite a surprise, quite novel to me. Having a relatively small closed loop versus one big loop that has to go outside seems like a very big tradeoff, with a somewhat material and space intensive demand (a rack with 6x CDUs), but the fine grained control does seem obviously sweet to have. I wish there were a little more justification for the use of heat exchangers! The way water is distributed within the server is also pretty amazing, with each server having it's own "bus bar" of water, and each chip having it's own active electro-mechanical valve to control it's specific water flow. The TPUv3 design where cooling happens serially, each chip in sequence getting hotter and hotter water seems common-ish, where-as with TPUv4 there's a fully parallel and controllable design. Also the switch from lidded chips to bare chips, with a cold plate that comes down to just above, channeling water is one of those very detailed fine-grained optimizations that is just so sweet.
ChuckMcM 1y ago
Much of the Google use of liquid chillers was protected behind NDAs as part of its "hidden advantage" with respect to the rest of the world. It was the secret behind really low PUE numbers.
- throwaway2037 1y ago
  Do we know if other hyperscalers also use liquid chillers to achieve very low PUE values? I think I saw photos from xAI's new data center and there was liquid cooling.
  - ChuckMcM 1y ago
    I don’t know of any Co-lo types that did it, Microsoft had some creative ideas about things like that but I don’t know if they brought any to fruition. I suspect there are still opportunities there.
bri3d 1y ago
I don't think this comment is accurate based on the article, although you cite personal experience elsewhere so maybe your project wasn't the one that's documented here? > What Google is doing is using the huge chillers that would normally be cooling the air in the facility to cool water which is directly pumped into every server. From the article: > CDUs exchange heat between coolant liquid and the facility-level water supply. Also, I know from attaching them at some point that plenty of mainframes used this exact same approach (water to water exchange with facility water), not water to air to water like you describe in this comment and others, so I think you may have just not had experience there? https://www.electronics-cooling.com/2005/08/liquid-cooling-is-back/ https://www.electronics-cooling.com/2005/08/liquid-cooling-i... contains a diagram in Figure 1 of this exact CDU architecture, which it claims was in use in mainframes dating back to 1965 (!). I also don't think "This eliminates ANY air based transfer besides the chiller tower." is strictly true; looking at the photo of the sled in the article, there are fans. The TPUs are cooled by the liquid loop but the ancillaries are still air cooled. This is typical for water cooling systems in my experience; while I wouldn't be surprised to be wrong (it sure would be more efficient, I'd think!), I've never seen a water cooling system which successfully works without forced air, because there are just too many ancillary components of varying shapes to successfully design a PCB-waterblock combination which does not also demand forced air cooling.
- liquidgecka 1y ago
  > > CDUs exchange heat between coolant liquid and the facility-level water supply. Oh interesting I missed that when I went through in the first pass. (I think I space bared to pass the image and managed to skip the entire paragraph in between the two images so that’s on me. I was running off an informal discussion I had with a hardware ops person several years ago where he mentioned a push to unify cooling and eliminate thermal transfer points since they were one of the major elements of inefficiency in modern cooling solutions. By missing that as I browsed through it I think I leaned too heavily on my assumptions without realizing it! Also, not all chips can be liquid cooled so there will always be an element of air cooling so the fans and stuff are still there for the “everything else” cases and I doubt anybody will really eliminate that effectively. The comment you quoted was mostly directed towards the idea that Cray-1 had liquid cooling, it did, but it transferred to air outside of the server which was an extremely common model for most older mainframe setups. It was rare for the heat to be kept liquid along the whole path.
jwr 1y ago
> they are direct cooling the servers with chillers that are outside of the facility That is exactly what the Cray Y-MP EL that I worked with in the 90s/2000s did.

ambicapter 1y ago

So every time they plug in a server they also plug in water lines?

jedberg 1y ago
Looks like it. New server means power, internet, and water.
- fudgy73 1y ago
  just like humans.
liquidgecka 1y ago
[I am not a current Google Employee so my understanding of this is based on externally written articles and “leap of faith” guestimation] Yes. A supply and return line along with power. Though if I had to guess how its setup this would be done with some super slick “it just works” kind of mount that lets them just slide the case in and lock it in place. When I was there almost all hardware replacement was made downright trivial so it could just be more or less slide in place and walk away.
- nielsbot 1y ago
  Maybe similar to a gasoline hose breakaway https://www.opwglobal.com/products/us/retail-fueling-products/standard-dispensing-equipment/swivels-and-breakaways/reconnectable-breakaways https://www.opwglobal.com/products/us/retail-fueling-product...
- michaelt 1y ago
  Interestingly, entire supercomputers have been decommissioned [1] due to faulty quick disconnects causing water spray. So you can get a single, blind-mating connector combining power, data and water - but you might not want to :) [1] https://gsaauctions.gov/auctions/preview/282996 https://gsaauctions.gov/auctions/preview/282996
- semi-extrinsic 1y ago
  https://amphenol-industrial.com/products/liquid-cooling-systems/ https://amphenol-industrial.com/products/liquid-cooling-syst...
  - tesseract 1y ago
    CPC is in this market too https://www.cpcworldwide.com/Liquid-Cooling/Products/Blind-Mate https://www.cpcworldwide.com/Liquid-Cooling/Products/Blind-M... Non-spill fluid quick disconnects are established tech in industries like medical, chemical processing, beverage dispensing, and hydraulic power, so there are plenty of design concepts to draw on.
- scrlk 1y ago
ajb 1y ago
I remember reading somewhere that they don't operate at the level of servers; if one dies they leave it in place until they're ready to replace the whole rack. Don't know if that's true now, though. It does sound like connections do involve water lines though. As they are isolating different water circuits, in theory they could have a dry connection between heat exchanger plates, or one made through thermal paste. It doesn't sound like they're doing that though.
- cavisne 1y ago
  Definitely not true for these workloads. TPUs are interconnected, one dying makes the whole cluster significantly less useful.
- liquidgecka 1y ago
  It has not been true for a LONG time. That was part of Google early “compute unit” strategy that involved things like sealed containers and such. Turns out that’s not super efficient or useful because you leave large swaths of hardware idle. In my day we had software that would “drain” a machine and release it to hardware ops to swap the hardware on. This could be a drive, memory, CPU or a motherboard. If it was even slightly complicated they would ship it to Mountain View for diagnostic and repair. But every machine was expected to be cycled to get it working as fast as possible. We did a disk upgrade on a whole datacenter that involved switching from 1TB to 2TB disks or something like that (I am dating myself) and total downtime was so important they hired temporary workers to work nights to get the swap done as quickly as possible. If I remember correctly that was part of the “holy cow gmail is out of space!” chaos though, so added urgency.
  - throwaway2037 1y ago
    > part of the “holy cow gmail is out of space!” chaos This sounds like an interesting story. Can you share more details.
  - Cthulhu_ 1y ago
    I'd love to work in a datacenter at that scale sometime, it sounds like it's like working in a warehouse where you get a list of orders, servers to remove and pick up, but at the scales of the Googles et al, that's hundreds of server replacements a day, and production lines of new servers being built and existing ones being repaired or decommissioned. It's a fascinating industry, but only in my head as the only info you get about it is carefully polished articles and the occasional anecdote on HN, which is also carefully polished due to NDAs.
jayd16 1y ago
Maybe we can declutter things if they get PWoE(power and water over ethernet) or just a USB-W standard.
- Nition 1y ago
  It worked for MONIAC.
- taneq 1y ago
  Great, now I’ll have to figure out if my USB Hi-Flow cable is slowing down my USB Full-Flow drink bottle refilling.
  - toast0 1y ago
    It's not. Full flow is less than hi-flow... Your bottle might slow down other fills on the bus though :p
- Cthulhu_ 1y ago
  Related, I vaguely recall a concept from a Tesla charger for semis that proposed a charging cable with active coolant flowing through it as well to keep the wire cooled. edit: https://www.teslarati.com/tesla-liquid-cooled-supercharger-cables/ https://www.teslarati.com/tesla-liquid-cooled-supercharger-c...
  - jayd16 1y ago
    This is why I risk jokes on HN. Normally they're frowned upon but it's worth it to find interesting tech that meets the eccentric idea.
Hilift 1y ago
And a 12V battery.

nitwit005 1y ago

This was before I was born, so I'm hardly an expert, but I've heard of feeding IBM mainframes chilled water. A quick check of wikipedia found some mention of the idea: https://en.wikipedia.org/wiki/IBM_3090 https://en.wikipedia.org/wiki/IBM_3090

ChuckMcM 1y ago
When our mainframe in 1978 sprung a leak in its water cooling jacket it took down the main east/west node on IBMs internal network at the time. :-). But that was definitely a different chilling mechanism than the types Google uses.
jauntywundrkind 1y ago
Having to pre chill water (via a refrigeration cycle) is radically less efficient than being able to collect and then disperse heat. It generates considerably more heat ahead of time, to deliver the chilled water. This mode of gathering the heat and sending it out, dealing with the heat after it is produced rather than in advance, should be much more energy efficient. I don't know what surprises me about it so much, but having these rack-sized CDU heat-exchangers was quite a surprise, quite novel to me. Having a relatively small closed loop versus one big loop that has to go outside seems like a very big tradeoff, with a somewhat material and space intensive demand (a rack with 6x CDUs), but the fine grained control does seem obviously sweet to have. I wish there were a little more justification for the use of heat exchangers! The way water is distributed within the server is also pretty amazing, with each server having it's own "bus bar" of water, and each chip having it's own active electro-mechanical valve to control it's specific water flow. The TPUv3 design where cooling happens serially, each chip in sequence getting hotter and hotter water seems common-ish, where-as with TPUv4 there's a fully parallel and controllable design. Also the switch from lidded chips to bare chips, with a cold plate that comes down to just above, channeling water is one of those very detailed fine-grained optimizations that is just so sweet.

ChuckMcM 1y ago

Much of the Google use of liquid chillers was protected behind NDAs as part of its "hidden advantage" with respect to the rest of the world. It was the secret behind really low PUE numbers.

throwaway2037 1y ago
Do we know if other hyperscalers also use liquid chillers to achieve very low PUE values? I think I saw photos from xAI's new data center and there was liquid cooling.
- ChuckMcM 1y ago
  I don’t know of any Co-lo types that did it, Microsoft had some creative ideas about things like that but I don’t know if they brought any to fruition. I suspect there are still opportunities there.

bri3d 1y ago

I don't think this comment is accurate based on the article, although you cite personal experience elsewhere so maybe your project wasn't the one that's documented here? > What Google is doing is using the huge chillers that would normally be cooling the air in the facility to cool water which is directly pumped into every server. From the article: > CDUs exchange heat between coolant liquid and the facility-level water supply. Also, I know from attaching them at some point that plenty of mainframes used this exact same approach (water to water exchange with facility water), not water to air to water like you describe in this comment and others, so I think you may have just not had experience there? https://www.electronics-cooling.com/2005/08/liquid-cooling-is-back/ https://www.electronics-cooling.com/2005/08/liquid-cooling-i... contains a diagram in Figure 1 of this exact CDU architecture, which it claims was in use in mainframes dating back to 1965 (!). I also don't think "This eliminates ANY air based transfer besides the chiller tower." is strictly true; looking at the photo of the sled in the article, there are fans. The TPUs are cooled by the liquid loop but the ancillaries are still air cooled. This is typical for water cooling systems in my experience; while I wouldn't be surprised to be wrong (it sure would be more efficient, I'd think!), I've never seen a water cooling system which successfully works without forced air, because there are just too many ancillary components of varying shapes to successfully design a PCB-waterblock combination which does not also demand forced air cooling.

liquidgecka 1y ago
> > CDUs exchange heat between coolant liquid and the facility-level water supply. Oh interesting I missed that when I went through in the first pass. (I think I space bared to pass the image and managed to skip the entire paragraph in between the two images so that’s on me. I was running off an informal discussion I had with a hardware ops person several years ago where he mentioned a push to unify cooling and eliminate thermal transfer points since they were one of the major elements of inefficiency in modern cooling solutions. By missing that as I browsed through it I think I leaned too heavily on my assumptions without realizing it! Also, not all chips can be liquid cooled so there will always be an element of air cooling so the fans and stuff are still there for the “everything else” cases and I doubt anybody will really eliminate that effectively. The comment you quoted was mostly directed towards the idea that Cray-1 had liquid cooling, it did, but it transferred to air outside of the server which was an extremely common model for most older mainframe setups. It was rare for the heat to be kept liquid along the whole path.

jwr 1y ago

> they are direct cooling the servers with chillers that are outside of the facility That is exactly what the Cray Y-MP EL that I worked with in the 90s/2000s did.