23 ms·
Nvidia adds liquid-cooled GPUs
- 2sk21 4y agoReminds me of the old IBM mainframes (until the 1990s) that were liquid cooled. They had an ingenious approach called TCM (Thermal Conduction Module) with the silicon chips directly attached to a cooled substrate.
- broodbucket 4y agoPretty sure IBM mainframes are still watercooled, though I don't know the specific method
- MikusR 4y agohttps://youtu.be/ZDtaanCENbc?t=116 https://youtu.be/ZDtaanCENbc?t=116 self contained liquid cooling.
- leoc 4y agoEEVBLOG tore one down: https://youtu.be/xQ3oJlt4GrI https://youtu.be/xQ3oJlt4GrI .
- usedgraphiccard 4y agoBtw: Has someone yet 'fiddeled' out how to add a keyboard directly to the graphics-card ? There are Ports, and RAM...(huge RAM)...so ? OT... (-;
- wongarsu 4y agoNvidia GPUs have a couple of microcontrollers for various management and ancillary tasks, you could probably take over one of those and take keyboard input from there? (search [1] for Falcon, those are what Nvidia is now moving to RISC-V for). Not sure what use that would be though 1: https://buildmedia.readthedocs.org/media/pdf/envytools/latest/envytools.pdf https://buildmedia.readthedocs.org/media/pdf/envytools/lates...
- beninsydney 4y agoI really like the Noctua cards they're collaborating on, it's not really necessary to go water-cooled once you've got a 4ish slot card/heatsink with quiet 120mm fans! https://noctua.at/en/asus-geforce-rtx-3080-noctua-edition-graphics-card https://noctua.at/en/asus-geforce-rtx-3080-noctua-edition-gr...
- willis936 4y agoIt's not really necessary to go 4-slot air cooler when you've got water cooling.
- moffkalast 4y agoWater cooling is just a way to transfer the heat to a larger heatsink somewhere else. It doesn't magically absorb the heat but just adds complexity and overhead. If there's a way to make a direct heatsink work, then that's the more convenient option every time. For datacenters it would make more sense to make it all water cooled though, since you can likely have one central loop connected to an HVAC unit or something.
- mcronce 4y agoIt also removes a big chunk of metal and fans taking up space on your motherboard, covering other slots. It adds complexity, not really sure what the "overhead" here is that you're referring to.
- willis936 4y agoWater cooling is more than that. Heatpipes have much lower transfer rates than pipes with water flowing. Pound for pound you dissipate a lot more heat with water being pumped through the rad. Ignoring that, putting the radiator anywhere other than the precious motherboard real estate has a lot of value. It's such a waste to hang a 10 pound piece of copper over all of those fast, unused PCIe lanes.
- CoastalCoder 4y ago
- zcw100 4y agoThis makes very little sense. How exactly does water cooling contribute to power efficiency? Once you get that heat off the chip it has to go somewhere and what is the complexity of running water cooling hoses all over the datacenter. How efficient is it if you get a leak even 1% of the time?
- DiabloD3 4y agoIt makes the cooling part a lot more efficient. You're right about the leak issue, this is unsolvable and banned in most datacenters. A lot easier to go hyperdense and go with chilled rear doors and/or in row cooling if this becomes your problem.
- deleted 4y ago[deleted]
- amelius 4y agoYes, the article seems very paradoxical.
- jabl 4y agoTransporting X Joules of heat around by pumping water is definitely a lot more efficient than transporting the same heat energy around by blowing hot air. So pumping losses are much smaller.
- latchkey 4y agoRack level heat exchangers... https://www.youtube.com/watch?v=4Np1HnWiHb4 https://www.youtube.com/watch?v=4Np1HnWiHb4 https://schroff.nvent.com/sites/g/files/hdkjer281/files/acquiadam/2020-11/NVENT-WPCS-H85178-WPHybridCooling-USEN.pdf https://schroff.nvent.com/sites/g/files/hdkjer281/files/acqu...
- dangus 4y agoData centers need this for space constraints, but if you're just the average Joe gamer I encourage you to think twice before considering liquid cooling (especially a custom loop). Why do I say this? Because...my goodness, it gets expensive, and you're really not going to get much more performance or even noise reduction out of it (and like...don't you already wear a headset while you game?) I've never understood the market for CPU AIOs especially. They're just so expensive compared to fitting a giant heatsink and fan on there, which will be quite quiet if it's large enough.
- Filligree 4y agoI've built systems with air cooling, with water cooling, and I've also bought an air-cooled prebuilt. My goal, in each case, was to minimize noise. What I've taken away from that is: - It's perfectly possible to build a quiet, air-cooled, relatively cheap system. - But you probably don't have the skill for that. It's a complex task, and requires either good intuition or an entire engineering degree. - Building a quiet water-cooled desktop is relatively easy, because you can oversize the cooling massively. So would I recommend water-cooling? My goodness no. It's a lot of work, it's expensive, and in the end the pre-built was marginally quieter. But if the comparison is against a self-built air-cooled one, then water cooling can indeed come out on top.
- dangus 4y ago> But you probably don't have the skill for that. It's a complex task, and requires either good intuition or an entire engineering degree. That’s just ridiculous, come on. You google some reviews of some parts or watch some Gamers Nexus videos and that’s all you have to do.
- dangus 4y agoI’ll add on top of this: some of the air cooled solutions are stupidly cheap. The Cooler Master Hyper 212 Evo has been on the market forever and it’s dirt cheap, good quiet performance, upgradable to any fan you damn please. It’s really as simple as a big-ass hunk of metal, it’s hard to go wrong. As far as graphics card noise you can usually buy something off the shelf and undervolt it in MSI Afterburner…or just buy a very large three fan mid-range graphics card where the cooling package is designed for something that draws way more power.
- politelemon 4y agoMinor, that should be 66% fewer in the chart, not "less". So question, how do these compare in terms of cost against the non liquid variants? I've been buying the Evga Hybrid variants of Nvidia GPUs, and they are not cheap at all, more on the premium side. That's the only cost reference I've got.
- Gigachad 4y agoArticle says closed loop cooling so I imagine it’s just a big scale version of pc cooling but the radiators vent outside. But I wonder if you could pump sea water in to a datacenter and cool using an open loop. Perhaps salt and dirt would become too much of a problem.
- Scandiravian 4y agoIn Denmark it's used to warm nearby homes. The heat is valuable and literally throwing it into the sea seems like a big waste.
- marvin 4y agoVery geography dependent, but e.g. in the Nordics waste heat is a resource. This is why we burn our combustible trash instead of putting it in landfills. Unfortunately, for most of the year the same wouldn’t we viable in California, and I guess the investment costs make the whole thing unprofitable.
- bozhark 4y agoMaybe you could use sea water to cool a coolant?
- kedikedi 4y agoThat’s how it is done in, for example, boats as well as other places where you want to extract heat to a lake or to a sea. You have your typical nice and clean coolant circuit that circulates through the engine, and then through a heat exchanger, you give that heat out to the sea water. This way the heat exchanger becomes somewhat sacrificial and you replace only that if it gets destroyed. And you don’t have to harden every single surface for sea water and all the creatures that come with it.
- jabl 4y agoBoat engines with direct seawater cooling are definitely a thing. Usually only on the cheapest and nastiest engines, better ones definitely have a separate heat exchanger.
- smileysteve 4y agoSo this saves energy because cooling with water is more efficient than with air. The Data center also needs to add water cooling for all power supplies and chips for maximum efficiency.
- zozbot234 4y agoWater cooling is hugely problematic because the cooling system will eventually spring a leak, and having water spraying/dripping all over your expensive server hardware is not anyone's idea of fun. That's why data centers use air cooling and a separate HVAC.
- newsclues 4y agoData centre water cooling is just like car water cooling and it’s absolutely a thing
- BeefWellington 4y agoSure but notably car water cooling, as robust as it is, does eventually spring leaks. I think it's a question of whether the efficiency tradeoff is worth the maintenance tradeoff (which we probably won't know since there's not a ton of these kinds of solutions in use). With air cooling it's easier since the entire medium around the servers is the fluid. With liquid cooling, short of doing a bath setup (which doesn't appear to be what's proposed here) I'm not convinced it's going to work well; Just imagine 2-4 pipes running to every 2U server on a rack. IMO, we shouldn't be against trying an idea again if it seems like on paper it could make a lot of sense. There's a lot to like about the potential of this solution; improved efficiency, density, etc. Heck, you might even make a fun side project of trying to reclaim that waste heat and turn it into electrical generation. I just think the plumbing alone is likely to make it cost ineffective compared to existing air systems and then add quite a bit of potential maintenance headache.
- smileysteve 4y ago> notably car water cooling, as robust as it is, does eventually spring leaks. Internal combustion engine cooling is not the same application as industrial computing. Differences include - ICE Cooling relies on pressure (boiling if released to atmospheric pressure - Maintenance (Schedules and Leak Detection); Most modern cars go a lifetime, if not 100k miles with no cooling system maintenance; if preventative maintenance is performed (replacement of radiator, pressure tank and/or rubber hosing regularly)and the system is monitored for level, repairs can be pre-empted. (very rarely does the leak happen in the block)) - Vibration and Environment (including cost or effort of maintenance)
- rocqua 4y ago> If you switched all the CPU-only servers running AI and HPC worldwide to GPU-accelerated systems, you could save a whopping 11 trillion watt-hours of energy a year. That’s like saving the energy more than 1.5 million homes consume in a year. That is a bold claim. It seems to me they simply assume all necessary code would trivially be converted to run on GPUs.
- sprkwd 4y agoIf everything is GPUs, then nothing is GPUs?
- AYBABTME 4y agoJust make that a General Processing Unit.
- dotnet00 4y agoA lot of HPC code is by its very nature pretty parallelizable (given that supercomputers these days are often just a large cluster of typical datacenter CPUs/GPUs with very fast interconnects), so is theoretically portable to GPUs for some gain. The hangups mostly seem to be in that a lot of scientific code hasn't been written with GPU style parallelization in mind, so updating the code takes time. That said, since a lot of newer supercomputers are including a lot of GPUs, there is a lot more incentive for developers to attempt to port at least the parts that can benefit from GPUs. From my (admittedly limited) experience, the biggest hangup is convincing scientists that GPUs aren't just another tech fad and it's worth the effort to port. There also seems to be a bit of uncertainty over which platform to trust, OpenCL support has been shaky on most vendors and is fairly primitive in terms of features, CUDA is tied to one company and AMD doesn't have a great record of supporting HIP across their product stack, so choosing any one is a big decision (comparable to choosing to use C/C++ or Python, which are effectively guaranteed to be supported on every supercomputer for the next decade or two).
- kxyvr 4y agoI work as a computational mathematician. Part of the problem on my end is that while the libraries have gotten better, they have not contained the operations that I need for good algorithms in my field. As an example, when GPUs first came onto the field, cublas computed a subset of the BLAS routines, which are pretty fundamental for putting together known, good algorithms. Eventually, that got better, but then the operations needed to fit onto a single GPU. Eventually, cublas-xt came onto the market and that allowed parallelization across GPUs, which was great. However, it still didn't have all of the needed matrix operations. At the moment, I'm not sure where it currently is. Of course, matrix algebra was only one of the problems. BLAS supports the writing of LAPACK routines, which tend to deal more with dense factorizations and eigenvalue problems. I believe more factorizations have been implemented recently, but I'm not currently up to date. Nevertheless, that was absolutely a bottle neck for the longest time. Yes, iterative methods don't need factorizations, but a good fraction of the preconditioners do. Then, of course, this speaks nothing of the sparse linear algebra problem. There's multiple ways to do it, but many of the good sparse factorization routines need dense factorization routines, so these needed to come onto the market first and that took time. And, to be clear, I know that there are multiple, good teams working on this. It takes time. And, there's a huge number of operations. If you're bored, go look at the manual for Intel's MKL and see the number and variety of operations that it provides. Those operations are there because people like me need them to do our job. I'll also agree that the kinds of operations we use to do our job will evolve over time with hardware. However, matrix algebra, factorizations, and eigenvalue problems lie at the very core of applied mathematics and expecting the mathematics to rework the last several hundred years of practice to accommodate the lack of tooling from the GPU manufacturers isn't realistic either. Anyway, if someone knows the current state of what's possible, I'd love to hear. Selfishly, what it really boils down to is what operations (algebra, factorization, or eigenvalue), how big (how much memory or on multiple GPUs), and dense or sparse.
- oblak 4y agoInteresting how ever increasing power and cooling requirements somehow equate to sustainable, efficient computing. nvidia hardware really is magical
- natly 4y agoThey're not really wrong. Imagine the vast vast amount of energy that'd be required to power the worlds compute using 90s hardware.
- CoastalCoder 4y agoYeah, I didn't follow their logic either. I can only think of two legit factors: 1) Somehow the energy needed to run a chilled water system is lower than a chilled air system. 2) Chilled water might practically allow the boards to run cooler, so their electrical resistance would be lower, making them more power efficient.
- AYBABTME 4y agoIf you can centralized the cooling in a few higher efficiency locations, and minimize heat transportation related power usage, you'll save on power.
- hulitu 4y agoI would say that "ever increasing power and cooling requirements somehow equate" to bad hardware development. Why optimise when you can cool it and the client pays the electricity bill.
- oblak 4y ago
- Dasateam 4y agoBecause the increase in power consumption is lower than the increase in performance. A given workload takes less energy to complete.
- 4y ago