5 ms·
Because it’s complex. Even more complex than “engineered” air. You need two circuits, and a CDU between them. Coolants needs maintaining. You add antifreeze, b
by bayindirh 3y ago
Because it’s complex. Even more complex than “engineered” air.
You need two circuits, and a CDU between them. Coolants needs maintaining. You add antifreeze, biocides, etc.
Air is brute force. It cools everything it touches. Liquid cooling is serialized in a node. Two sockets? Second will be hotter. HBA not making good contact? It’ll overheat.
You add extensive leak detection subsystems, the amount of coolant moving in your primary circuit becomes massive.
Currently you can remove 97% of the heat via liquid (including the PSUs), and it’s cheaper to do so than air, but it’s not “rails, screws, cables, power on”. Air cooled systems can be turned on in a week. Liquid cooled ones take a month.
However, using liquid is mandatory after some point. You can’t cool systems that dense and under that load with air. They’ll melt.
- gopher_space 3y agoWhat's this all look like without an atmosphere?
- eutropia 3y agoWorse, heat dissipation is a major constraint for spacecraft and satellites because you can only radiate heat away as infrared photons.
- gopher_space 3y agoCouldn't you heat up and poop out blocks of material all day long, or is that starting down a "solution creates more heat than it dissipates" path?
- taneq 3y agoDoesn’t have to be infrared but yeah, space isn’t “cold” so much as it’s an insulator.
- _a_a_a_ 3y agoat ~2 kelvin I'd have thought you can radiate away a truckload of heat surely
- zmgsabst 3y agoYou can radiate easily. But not convect. Hence why it’s much, much harder than removing heat on Earth.
- _a_a_a_ 3y agoOf course you're not convecting but if you are radiating from a hot body into an ambient two Kelvin then you are going to lose heat really, really fast. IIRC heat loss by black body radiation into its surroundings is proportional to the fourth power of the temperature difference between the body and surroundings (from memory, and going back a very long way, so maybe incorrect).
- amluto 3y ago> proportional to the fourth power of the temperature difference between the body and surroundings Almost. It’s proportional [0] to T_hot^4 - T_cold^4. For a 100C surface with emissivity 1, that’s about 1kW/m2 if there is no radiation coming back, which really isn’t very high. You cannot cheat this with fancy folded-up radiating surfaces (it’s thermodynamically impossible, and the actual mechanism that kills it is one fin of the heatsink radiating right at the next one). So cooling in space is hard. You’re not getting GPU-like power densities without a physically immense radiating surface extending way past those GPUs. [0] Caveat: emissivity can depend on wavelength, and the law holds independently at each wavelength. So this can introduce interesting effects, which is how all the fancy prototype roof-cooling materials work, and it’s also related to how “spectrally selective” windows and window films work.
- _a_a_a_ 3y agoThanks, you clearly know your subject. I must say that without doubting your figures, I am astonished at how little heat can be lost radiatively. Given the 4th powers I assumed it must be vastly higher, but clearly not, much against my intuition. Thanks for a good answer!
- uticus 3y agoAmazing considering how much heat travels from Sun (and punches through atmosphere) to Earth surface. Didn’t realize there was that much of an insulation property.
- fecal_henge 3y agoYou just need to radiate in the visible spectrum then the problem will be much reduced.
- magicalhippo 3y ago> Liquid cooling is serialized in a node. Two sockets? I've seen tests done on heavy PC loops (ie multi-GPU) both high-flow and low-flow, as well as on car engines, in different coolant flow configurations. The results from all of those are that the water doesn't rise meaningfully in temperature between components. Unless I did my back-of-the-napkin math wrong, this seems reasonable. If you have a single 10mm ID pipe going through a 1U server and up to the next, then for a full 42U rack you have about 1.7kg of water going through the servers. If the flow rate is about 1s per server (so 42 seconds for the full rack) and each 1U server dumps 500W of energy into the water, there should be just a 3 degree C difference in the water temperature between the first and the last server.
- bayindirh 3y agoIn our system every node gets inlet water at the same temperature via parallel piping, but when it’s in node, it goes through processors first, then RAM, then PCIe and disks. Delta T between two sockets is 5 degrees C, and the delta T between input and output is around 15-18 C depending on load.
- menaerus 3y agoFirst, thanks for sharing these details, I find them fascinating because they are not so common to be read or heard about. > Delta T between two sockets is 5 degrees C And secondly, ~5-10 degrees is what I see on my dual-socket workstation, and have been wondering about this delta ever since the first day I started monitoring the temperatures. At first, I thought that the heat sink wasn't installed properly but after reinstalling it the delta remained. Since I didn't notice any CPU throttling or whatsoever I figured it's "normal" and ignored it.
- bayindirh 3y agoHey, no worries. Using one is equally fascinating as much as reading about it. It feels like a space shuttle, so different, yet so enjoyable. I mean, water travels from one socket to another, so one processor adds heat equal to 5 degrees C under nominal load. The second socket doesn’t complain much, but this is enormous amounts of heat transferred in a such quick pace.
- is_true 3y agoIs rising air's water content (humidity) worth it? Humid air can "store" more heat. I guess it could be bad past some %, but there's a probably a point where it's worth it.