5 ms·
Liquid and immersion is the new cool at Supercomputing '22
- aidenn0 4y agoI want to try immersing a RPI in mineral oil. It should fit in a metal pint paint can; epoxy around the ports and it should at least only leak slowly when tipped over.
- Dylan16807 4y agoThat should work, but I think for bulky cooling I'd just stick the pi to a half inch thick square of aluminum.
- lost_tourist 4y agodo it!
- ramshanker 4y ago100%. It has to be that easy. I hope soon.
- 4str0n0mer 4y agoThis pops up every few years. I can remember building a liquid cooled LAN-party rig at some point. But imho especially immersion cooling is a dead end. Contact problems with cabling lower reliability because the isolating coolant creeps between the metal surfaces. Maintenance is unbelievably convoluted, not only do you have to drain all the fluid out to pull a machine, you will need to clean each and every plug, socket and pin you touch because the contact surfaces will be coated with isolating coolant. This could only be fixed with a complete redesign of everything, so that all or at least most contacts are "dry". Using fluid-tight plugs is very expensive, and anyways, most ideas there won't work for important components such as CPU and RAM. Liquid cooling will fare better, because you can strike a balance between high-power components like CPU and GPU where you slap a liquid cooler on. And low-power components like most of the board you just cool by air as usual. But even there, maintenance is somewhat of a hassle, in addition to all the cabling you also need to deal with piping, leaks, drippage (which aren't supposed to happen, but trust me, they do). I think the only field where this is really going to be used is HPC for applications where floorspace is limited. E.g. if your network cabling must be within a certain length limit of your central network node, so you need to cram all your racks for the supercomputer you are building within some 20m circle or the like. Less top-of-the-line HPC and datacenters are better served by just limiting power density and continuing air cooling as usual. Datacentre floorspace is expensive, but not _that_ expensive, so you will always fare better (and usually more environment-friendly) with air cooling.
- Scandiravian 4y agoI think this is a nice summary. As a point for water cooling, it's easier to use the generated heat; it can for instance be used to heat nearby buildings. With air cooling this is much more difficult. I think watercooling will become dominant in new data centers as environmental laws sets stricter requirements for heat waste.
- bo0tzz 4y ago> With air cooling this is much more difficult To my understanding, DCs use air conditioning systems for cooling. These are already heat pumps - why would it be more difficult to just pump that heat into district heating instead of outside air?
- 4str0n0mer 4y agoBecause the temperature spread a given heat pump uses isn't really suitable for district heating, your output water would only be in the 50 to 60°C range at most, too low for typical district heating systems. One can get around that with multi-stage heat pumps or different coolants, but both are expensive investments typically not viable for existing systems.
- nine_k 4y agoIf it's not enough for district heating because of the distance and heat losses, it could be enough for a greenhouse right next to a DC. Economically the greenhouse would have a distinct advantage of using free heat, and already having enough electricity wired nearby. This, of course, only makes sense during cold seasons, or somewhere like Alaska or Sweden. During hot months, 40-50°C of heat seem completely useless. Maybe next to a sea such waste heat could help evaporatively desalinate seawater, while cooling back to reusable temperatures.
- 4str0n0mer 4y agoYes, heat reuse is easier with water cooling since the losses are smaller, but even air cooling will produce warm water in the end. There will be coming regulations requiring a certain amount of heat reuse, but in a first step, this probably won't force direct water cooling. For those regulations, there is also the important aspect of free cooling, i.e just using outside air as a medium or using an air-air heat exchanger like a Tokyo Wheel. Mandated heat reuse (without appropriate exceptions) would make those illegal, however in certain situations those are the best (environmentally and financially) methods of cooling. E.g. in a moderately hot summer you can use 20-30°C outside air without a problem, but if you would mandate heat reuse at that time, you would have lots and lots of excess heat on your hands with nobody buying it. So while some regulation will be coming, I hope and think that it wouldn't really issue a strict mandate to reuse heat and allow free air cooling, which would exclude direct water cooling.
- mprovost 4y agoWe had 8 systems in the Top 500 list at Weta Digital back in 2008 which were liquid cooled HP servers for rendering Avatar. We used one of the techniques mentioned in the article: liquid to liquid heat exchangers. Each rack was fully sealed with its own heat exchanger which pumped in cool water and sent off warm water to chillers on the roof. Actually the warm water condensing would attract seagulls in the winter which was a problem since we were right next to the airport. The datacentre was designed by an engineering firm which usually built dairy factories. Every once in a while something would go wrong with the cold water delivery system and the servers would overheat and shut down in about 30 seconds.
- deegles 4y agoLast time I looked into this the best available coolant was the 3M Novec line, but it was around $400 usd/gallon. Is it available anywhere else for less?
- deleted 4y ago[deleted]
- techdragon 4y agoAnd I could be wrong and this could be from a previous iteration of this “inert” liquid dialectic insulators chemical line from 3M but I believe some of them have at least in the past had a less the ideal toxicity profile or have had issues with the normal and inevitable slow chemical breakdown from ionising and cosmic radiation… producing notably toxic gasses you definitely wouldn’t want slowly building up unmonitored in an enclosed space for the years. Don’t get me wrong, the Novec line and it’s ancestors and siblings like the stuff that cooled the old Cray supercomputers… are the sort of Chemical wizardry that had chemical engineering on the prospective career list when I was in high school. I just remember seriously investigating using them for a project regardless of the price and coming across some sort of chemical safety red flag. So if anyone knows/recognises what I’m half remembering, feel free to share the details!
- c_o_n_v_e_x 4y agoWe used this stuff for a small pilot project: https://www.engineeredfluids.com/electrocool https://www.engineeredfluids.com/electrocool Pre-covid we paid roughly $225 for 20 liters, not bulk pricing. Not quite apples to apples as 3M Novec is a 2 phase coolant, we went with single phase.
- latchkey 4y agoI run large scale data centers. Crypto mining has been pushing a lot of this technology ahead as well [1] (yea, yea). Liquid cooling is nice on paper, but as others have noted, it is messy AF. Another huge issue that you don't normally think aobut is staffing... some data centers are in the middle of nowhere. It is hard enough to just fine skilled enough labor to work reliably... getting them trained up to do maintenance on wet systems is a whole another level of pain. It is great for small systems... not so good at scale. It is amazing how proper airflow with fans can solve most cooling needs. Even if the outside temps are high as well. [1] https://twitter.com/SenTedCruz/status/1564441153306406912 https://twitter.com/SenTedCruz/status/1564441153306406912
- homeland221 4y agoLet's be honest here..."...getting them trained up to do..."...probably doesn't exist these days. In my 3 decades of HR interviewing exits, near 19 out of 20 replied no training...heck some don't get any proper intro by their manager in charge. During interviewing, hiring managers already expect candidates are better than the one leaving complete with company internal workflow knowledge. During 2008, the expectation of resume thickness exceed a doctorate-master thesis.
- jjtheblunt 4y agoWasn't the Cray 2 liquid immersion cooled, 35 years ago, famously? https://en.wikipedia.org/wiki/Cray-2 https://en.wikipedia.org/wiki/Cray-2
- 20100thibault 4y agoA couple of interesting factors leading to liquid cooling in the commercial market (always been prominent in the top 500) and why I think this time it’s different. First, we are designing smaller transistor leading to leakage and thus increasing chip thermal design power (TDP). We are also using bigger chips and chiplets also increasing TDP. TDP on a GPU went from 200w to 1 kW in the past 5 years and even CPU are now close to 500w. At 25k a pop avoiding throttling due to heat restriction is a must and is becoming incredibly hard with air cooling as air is an amazing insulator. Second, networking is extremely expensive with the cost of top-end InfiniBand cable close to 1000$ a foot. This means you want to cram your CPU /GPU as close as possible to keep the connection electrical if possible and minimize optical cable lengths. This also decreases latency and increase cluster performance. On a 50 000 cable deployment, the saving can be … significant. Third, we are now seeing the emergence of foundational model that requires government lab levels of interconnected compute. This is a new paradigm in the private sector and consume power in the 20-30 MW range. An increase in efficiency of 10% (from 1.2 to 1.1 PUE) brings huge saving on the power cost side plus important environmental benefits. It also becomes much easier to recuperate heat. That’s why we’re seeing major announcement at OCP i.e. meta going the liquid cooling way. I personally believe that’s where things are going and we are building a facility to optimize for the above points: https://www.qscale.com/ https://www.qscale.com/
- jiggawatts 4y ago> InfiniBand cable close to $1000 a foot Wait... wat? I thought InfiniBand just used fibre!
- 20100thibault 4y agohttps://www.naddod.com/products/18224.html https://www.naddod.com/products/18224.html that's the 2018 200g standard at over 1000$ a meter. Can't find public pricing but the new 400/800 gb/s are more expensive!
- jiggawatts 4y agoThat price is 95% the endpoints, 5% the cable. The fibre in between the QSFP adapters on the end is practically free. The 30m pricing is almost the same: https://www.naddod.com/products/18231.html https://www.naddod.com/products/18231.html 1m $1,087 10m $1,124 30m $1,196 A bit of linear regression on the back of the envelope yields $3.76 per meter for the fibre, and $560 as the cost per QSFP adapter.
- web007 4y agoIt's absolutely delightful to see concepts come many full circles like this. I remember the Cray-2 cooling towers as the most futuristic computer in the world even in the 90s, and it inspired me to sink my old system in mineral oil in the early 00s. I did the research and found the stuff in the Cray towers (Fluorinert) was $many/gallon, and the smallest volume they sold it was something like 55gal drums. Mineral oil was $cheap enough, also inert, and so I got a 5 gallon tub of the stuff. I built a plexiglass case, sealed it with silicone caulk, overclocked the CPU, hooked up an industrial-strength aquarium circulation pump and... rapidly went nowhere with a disgusting, oily motherboard. And a ridiculously heavy case filled with, essentially, hazmat. It was a terrible idea back then and still seems bad today. I'll be interested to see what the next cycle brings, if it's back to spot cooling or if immersion really is the way of the future.
- 71a54xd 4y agoNot seeing anything mentioning 3m's improved Fluorinert phase change dielectric coolants? Fortunately I had the pleasure of using these coolants in high school when I was hired to help build a commercial scale ASIC farm in the midwest - we used about 200 gallons of 3m Novec 7000. Which is insane since that coolant cost around $180/gal even at a steep discount ($36,000 in total). And that's after the shipment arrived with 10% of the bottles "empty" since the container sat in a hot port - coolant simply slipped past gaskets on the glass bottles.