4 ms·

This pops up every few years. I can remember building a liquid cooled LAN-party rig at some point. But imho especially immersion cooling is a dead end. Contact

This pops up every few years. I can remember building a liquid cooled LAN-party rig at some point. But imho especially immersion cooling is a dead end. Contact problems with cabling lower reliability because the isolating coolant creeps between the metal surfaces. Maintenance is unbelievably convoluted, not only do you have to drain all the fluid out to pull a machine, you will need to clean each and every plug, socket and pin you touch because the contact surfaces will be coated with isolating coolant. This could only be fixed with a complete redesign of everything, so that all or at least most contacts are "dry". Using fluid-tight plugs is very expensive, and anyways, most ideas there won't work for important components such as CPU and RAM. Liquid cooling will fare better, because you can strike a balance between high-power components like CPU and GPU where you slap a liquid cooler on. And low-power components like most of the board you just cool by air as usual. But even there, maintenance is somewhat of a hassle, in addition to all the cabling you also need to deal with piping, leaks, drippage (which aren't supposed to happen, but trust me, they do). I think the only field where this is really going to be used is HPC for applications where floorspace is limited. E.g. if your network cabling must be within a certain length limit of your central network node, so you need to cram all your racks for the supercomputer you are building within some 20m circle or the like. Less top-of-the-line HPC and datacenters are better served by just limiting power density and continuing air cooling as usual. Datacentre floorspace is expensive, but not _that_ expensive, so you will always fare better (and usually more environment-friendly) with air cooling.

Scandiravian 4y ago
I think this is a nice summary. As a point for water cooling, it's easier to use the generated heat; it can for instance be used to heat nearby buildings. With air cooling this is much more difficult. I think watercooling will become dominant in new data centers as environmental laws sets stricter requirements for heat waste.
- bo0tzz 4y ago
  > With air cooling this is much more difficult To my understanding, DCs use air conditioning systems for cooling. These are already heat pumps - why would it be more difficult to just pump that heat into district heating instead of outside air?
  - 4str0n0mer 4y ago
    Because the temperature spread a given heat pump uses isn't really suitable for district heating, your output water would only be in the 50 to 60°C range at most, too low for typical district heating systems. One can get around that with multi-stage heat pumps or different coolants, but both are expensive investments typically not viable for existing systems.
    - nine_k 4y ago
      If it's not enough for district heating because of the distance and heat losses, it could be enough for a greenhouse right next to a DC. Economically the greenhouse would have a distinct advantage of using free heat, and already having enough electricity wired nearby. This, of course, only makes sense during cold seasons, or somewhere like Alaska or Sweden. During hot months, 40-50°C of heat seem completely useless. Maybe next to a sea such waste heat could help evaporatively desalinate seawater, while cooling back to reusable temperatures.
- 4str0n0mer 4y ago
  Yes, heat reuse is easier with water cooling since the losses are smaller, but even air cooling will produce warm water in the end. There will be coming regulations requiring a certain amount of heat reuse, but in a first step, this probably won't force direct water cooling. For those regulations, there is also the important aspect of free cooling, i.e just using outside air as a medium or using an air-air heat exchanger like a Tokyo Wheel. Mandated heat reuse (without appropriate exceptions) would make those illegal, however in certain situations those are the best (environmentally and financially) methods of cooling. E.g. in a moderately hot summer you can use 20-30°C outside air without a problem, but if you would mandate heat reuse at that time, you would have lots and lots of excess heat on your hands with nobody buying it. So while some regulation will be coming, I hope and think that it wouldn't really issue a strict mandate to reuse heat and allow free air cooling, which would exclude direct water cooling.
c_o_n_v_e_x 4y ago
> >so you will always fare better (and usually more environment-friendly) with air cooling. Sorry but this is not true. One of the key drivers of liquid cooling IS efficiency. Liquids are for more effective at heat transfer than air. Additionally, a liquid can carry a lot more heat. Yes, you can run higher density, you can also indirectly free cool servers year-round, even in tropical climates.
- lightedman 4y ago
  The problem with liquid cooling, ultimately, is you're still dumping to air. You're actually doing not much more with liquid cooling than just changing where the heat concentration is located. Unless your radiator is outside of the building itself, you're dumping that waste heat right back into the system over time.
  - c_o_n_v_e_x 4y ago
    >The problem with liquid cooling, ultimately, is you're still dumping to air. There's a lot of things you can do outside of a server chassis that you cannot do inside of a chassis. You can have heat exchangers with far more surface area than what you are volume constrained to inside of a chassis. You can run evaporative cooling towers. If you're in the right climate, you can even radiate that heat to space. >You're actually doing not much more with liquid cooling than just changing where the heat concentration is located. You're using a more efficient heat transfer medium which does a lot of things for you. You can downsize your chiller tonnage, reduce/eliminate CRAC/CRAH units, and all the UPS required to keep those systems running. All of this translates to CAPEX and OPEX savings. >Unless your radiator is outside of the building itself, you're dumping that waste heat right back into the system over time. No one is suggesting that you can plonk some servers into a tank of liquid and the thermal energy magically disappears. Of course, there has to be a thermodynamic "sink" that you are dumping to... the atmosphere, the ocean, the ground, space, etc. Liquid cooling is more efficient means of transferring heat from source to sink.
  - zmgsabst 4y ago
    Isn’t that the point? You use a liquid to transport the heat from denser compute (lowering costs), then run it through a high efficiency radiator (eg, drip/spray towers like with industrial AC). Your compute is denser and your interface with the air is more efficient.
jiggawatts 4y ago
What's your opinion of the Microsoft Ignite 2022 demo by Mark Russinovich of their next-gen immersion-cooled setup for Azure compute nodes? Link: https://youtu.be/PO5ijv6WDv0?t=370 https://youtu.be/PO5ijv6WDv0?t=370
- 4str0n0mer 4y ago
  Not good. As you can see, even the upper side connections are submerged, and plugs still look like standard components. Nothing to say of the backplane connectors on the lower side and RAM and CPU connections. They seem to just have taken a blade enclosure and flipped it backside up. So yes, it looks expensive, maybe impressive for some audience, but totally impractical even though they claim it to be a custom design.
  - jiggawatts 4y ago
    Your main concern seems to be with ongoing maintenance, but my understanding is that the "hyperscalers" like Azure don't do maintenance. They assemble everything once and then let failed components remain in-place and powered off. Google championed this over a decade ago. They figured that the cost of staff, spare parts, diagnostics, etc... outweighed the cost of the hardware. If you factor in the complexity of liquid cooling, it might not make sense to repair hardware even a single compute node costs north of $20K.

Scandiravian 4y ago

I think this is a nice summary. As a point for water cooling, it's easier to use the generated heat; it can for instance be used to heat nearby buildings. With air cooling this is much more difficult. I think watercooling will become dominant in new data centers as environmental laws sets stricter requirements for heat waste.

bo0tzz 4y ago
> With air cooling this is much more difficult To my understanding, DCs use air conditioning systems for cooling. These are already heat pumps - why would it be more difficult to just pump that heat into district heating instead of outside air?
- 4str0n0mer 4y ago
  Because the temperature spread a given heat pump uses isn't really suitable for district heating, your output water would only be in the 50 to 60°C range at most, too low for typical district heating systems. One can get around that with multi-stage heat pumps or different coolants, but both are expensive investments typically not viable for existing systems.
  - nine_k 4y ago
    If it's not enough for district heating because of the distance and heat losses, it could be enough for a greenhouse right next to a DC. Economically the greenhouse would have a distinct advantage of using free heat, and already having enough electricity wired nearby. This, of course, only makes sense during cold seasons, or somewhere like Alaska or Sweden. During hot months, 40-50°C of heat seem completely useless. Maybe next to a sea such waste heat could help evaporatively desalinate seawater, while cooling back to reusable temperatures.
4str0n0mer 4y ago
Yes, heat reuse is easier with water cooling since the losses are smaller, but even air cooling will produce warm water in the end. There will be coming regulations requiring a certain amount of heat reuse, but in a first step, this probably won't force direct water cooling. For those regulations, there is also the important aspect of free cooling, i.e just using outside air as a medium or using an air-air heat exchanger like a Tokyo Wheel. Mandated heat reuse (without appropriate exceptions) would make those illegal, however in certain situations those are the best (environmentally and financially) methods of cooling. E.g. in a moderately hot summer you can use 20-30°C outside air without a problem, but if you would mandate heat reuse at that time, you would have lots and lots of excess heat on your hands with nobody buying it. So while some regulation will be coming, I hope and think that it wouldn't really issue a strict mandate to reuse heat and allow free air cooling, which would exclude direct water cooling.

c_o_n_v_e_x 4y ago

> >so you will always fare better (and usually more environment-friendly) with air cooling. Sorry but this is not true. One of the key drivers of liquid cooling IS efficiency. Liquids are for more effective at heat transfer than air. Additionally, a liquid can carry a lot more heat. Yes, you can run higher density, you can also indirectly free cool servers year-round, even in tropical climates.

lightedman 4y ago
The problem with liquid cooling, ultimately, is you're still dumping to air. You're actually doing not much more with liquid cooling than just changing where the heat concentration is located. Unless your radiator is outside of the building itself, you're dumping that waste heat right back into the system over time.
- c_o_n_v_e_x 4y ago
  >The problem with liquid cooling, ultimately, is you're still dumping to air. There's a lot of things you can do outside of a server chassis that you cannot do inside of a chassis. You can have heat exchangers with far more surface area than what you are volume constrained to inside of a chassis. You can run evaporative cooling towers. If you're in the right climate, you can even radiate that heat to space. >You're actually doing not much more with liquid cooling than just changing where the heat concentration is located. You're using a more efficient heat transfer medium which does a lot of things for you. You can downsize your chiller tonnage, reduce/eliminate CRAC/CRAH units, and all the UPS required to keep those systems running. All of this translates to CAPEX and OPEX savings. >Unless your radiator is outside of the building itself, you're dumping that waste heat right back into the system over time. No one is suggesting that you can plonk some servers into a tank of liquid and the thermal energy magically disappears. Of course, there has to be a thermodynamic "sink" that you are dumping to... the atmosphere, the ocean, the ground, space, etc. Liquid cooling is more efficient means of transferring heat from source to sink.
- zmgsabst 4y ago
  Isn’t that the point? You use a liquid to transport the heat from denser compute (lowering costs), then run it through a high efficiency radiator (eg, drip/spray towers like with industrial AC). Your compute is denser and your interface with the air is more efficient.

jiggawatts 4y ago

What's your opinion of the Microsoft Ignite 2022 demo by Mark Russinovich of their next-gen immersion-cooled setup for Azure compute nodes? Link: https://youtu.be/PO5ijv6WDv0?t=370 https://youtu.be/PO5ijv6WDv0?t=370

4str0n0mer 4y ago
Not good. As you can see, even the upper side connections are submerged, and plugs still look like standard components. Nothing to say of the backplane connectors on the lower side and RAM and CPU connections. They seem to just have taken a blade enclosure and flipped it backside up. So yes, it looks expensive, maybe impressive for some audience, but totally impractical even though they claim it to be a custom design.
- jiggawatts 4y ago
  Your main concern seems to be with ongoing maintenance, but my understanding is that the "hyperscalers" like Azure don't do maintenance. They assemble everything once and then let failed components remain in-place and powered off. Google championed this over a decade ago. They figured that the cost of staff, spare parts, diagnostics, etc... outweighed the cost of the hardware. If you factor in the complexity of liquid cooling, it might not make sense to repair hardware even a single compute node costs north of $20K.