7 ms·
Intel plans immersion lab to chill its power-hungry chips
- deleted 5y ago[deleted]
- chroem- 5y agoThis is why I think silicon carbide based chips are going to be a huge deal: you can run them hot enough that you can actually run a heat engine off of the chip's waste heat to recuperate some of the electricity you spent on computation. Now if only I could figure out how to invest in companies developing this technology...
- picture 5y agoDoesn't silicon carbide have unusual properties for making digital circuits with? SiC diodes have forward voltages higher than regular Si, for example. And additionally, I'm not sure if you can get the same performance on SiC at super high temperature, as even the metalization need to be specialized to handle the heat without too much resistive voltage drop, etc
- AtlasBarfed 5y agoWasn't the original research push for CVD diamonds for CPUs so you could run them 10x hotter?
- azinman2 5y agoShouldn’t they instead figure out how to run cooler and with less power in general? That’s where it seems everyone else is going…
- booi 5y agoI need… more powahhh… No but seriously why don’t they build a 128-core atom server. That’s really all anybody wants. I don’t need the fastest most immersed cpu ever, just a bunch of decent ones at 30W or less.
- tadfisher 5y agoThat was Larrabee/Xeon Phi, was it not? Discontinued for lack of sales.
- sseagull 5y agoThat’s basically what I thought. However, IIRC they were marketed more towards high-performance computing (with avx-512). They were an uncomfortable middle ground though, between normal CPUs and GPUs. My benchmarks showed that there wasn’t much of an advantage over 20-ish normal xeon cores (for my HPC workloads). (Memory is a little fuzzy - that was 4-6 years ago).
- glowingly 5y agoWhile not exactly what you are looking for, Intel Snow Ridge is a continuation of their Atom-based (next to their line of Core-based) networking processors. 8-24 cores. https://www.intel.com/content/www/us/en/products/details/processors/atom/p.html https://www.intel.com/content/www/us/en/products/details/pro... Though, unless if you 100% need X86, there is the Ampere Altra 128 core Cortex-N1 chip.
- astrange 5y agoThat’d be a pretty unbalanced architecture - may use less rack space than 128 servers, but with only one servers’ worth of IO, network, PSUs, it’d be less reliable and maybe not even faster.
- booi 5y agoA single AWS graviton server is roughly 128 cores with, I believe, 2x or 4x 40gbit networking. It’s not crazy really.
- Dylan16807 5y agoOne core is probably not pushing more than 100mbit on average. Scaling that much networking up a hundredfold is child's play.
- icegreentea2 5y agoIntel (and everyone else) do work on improving compute efficiency. But as the article points out, if 40% of your DC's power consumption is in cooling, then you'd be foolish not to target that slice. Liquid and immersion cooling allows higher power density, which all things being equal (I know there's a lot of heavy lifting being done by this...) will be preferred. Why distribute your components over a rack if you could fit it into a single 4U board? Why distribute your components over an aisle if you could fit it into a rack?
- lumost 5y agoThe other advantage of power density is that it creates stronger convection currents. Whereas data centers have traditionally been actively cooled, it’s not unreasonable to imagine open air dcs with air channels to support convection.
- walrus01 5y agoit's sort of been done, though there are still a lot of active fans to move the hot air. https://www.google.com/search?channel=fs&client=ubuntu&q=chicken+coop+datacenter+ https://www.google.com/search?channel=fs&client=ubuntu&q=chi...
- voldacar 5y agoThere is only so much computation you can do per watt on current process nodes. To increase our computation per chip, which is the goal, we need to increase the amount of watts we consume per chip. The goal should be to make more powerful processors, not ones that do the same with less power.
- temac 5y ago> we need to increase the amount of watts we consume per chip. Not sure we need that, except in niches. At scale you often want at least some efficiency, which is certainly not max TDP per core (because the best efficiency point is with lower frequencies and higher width, not the max freq you can achieve). So remains the question of large number of cores, but at some point the area of silicon also goes stupid high. And you can put multiple packages, without sacrificing overall system density too much, and without departing from simpler, and probably lower TCO pollution. For small systems it depends, but you actually often have even more limited thermal budget, except again in niches if you are ready to tolerate the drawbacks (stupid power req it even becomes hard to have just a few machines on a basic electrical network in standard homes or offices, high noise under load, obviously high TDP so heating up a lot). But you have less space constraints so if you really want absurd systems you already can. So do we really need to e.g. double or triple the (electrical/thermal) power density at scale? Do we need 2 kW chips? Do we need to sacrifice the efficiency now, and increase the nominal consumption now, instead of waiting just a few years for node improvements? (And I could even ask: do we really need that much increase of processing power, shouldn't we start to optimise for the total ecological cost instead? and I've not tried to do some prospective in that area but maybe this would mean slowing down the processing power growth...)
- voldacar 5y ago>So do we really need to e.g. double or triple the (electrical/thermal) power density at scale? Will we ever be able to double or triple our (general purpose) compute per cpu any other way? Moore's law is essentially over. Node improvements aren't really happening outside of TSMC, which doesn't have enough manufacturing ability to supply everyone, and even then those node improvements are getting more and more incremental. And regarding power consumption, I think we really need to be consuming more energy across most sectors of human activity. The most likely explanation for the "great stagnation" is that our energy consumption has basically flatlined since the 70s. It appears that on a civilizational scale, reaching greater levels of development and expression simply requires more Joules. If you disagree, I highly recommend the book Where is my Flying Car?
- dev_tty01 5y agoAbsolutely. This is the wrong direction. Lots of power to run the chips, then lots more power to cool them. The ARM server market is eventually going to eat them alive if they can't rearchitect and seriously up their performance/watt game. I hope this is just an intermediate term solution.
- imtringued 5y agoWhat ARM server market? There is at best one ARM server chip per hyperscaler and most have zero ARM server chips. Conventional ARM server efforts failed because the ARM server vendors don't want to talk to mortals, only a very limited set of customers of which most intend to design their own chips and obsolete them. The amount of companies that failed to make their ARM servers available to the public is astonishing. Linus Torvalds said that ARM needs to be widespread on the desktop, because you need a critical mass of developers targeting ARM. ARM server vendors don't want that critical mass, they want special deals with a handful of big companies which is obviously doomed to fail.
- ece 5y agoWhat is the maximum performance % difference between optimizing for perf/$ and perf/watt? Sure, there are wafer scale chips now, but the TDP for a phone is still ~5W watts, average laptops have gone from ~15 to ~30W, and desktops from ~300-600W+. I suppose with Zen 4, there might actually be an apples to apples comparison baring ISA and uncore differences. If ADL is anything to go by I imagine performance will be within ~15% of each other, but with a ~30% price difference if you care about a more efficient and cooler running chip. Sure the efficiency gains add up, but so do the performance gains on the other side.
- wmf 5y agoWhat is the maximum performance % difference between optimizing for perf/$ and perf/watt? Alder Lake and M1 Pro are good demonstrations of those two approaches.
- ece 5y agoFor a crude comparison, the new framework with Alder Lake is ~20-30% less than the 14" MBPs with M1 Pro. If the uncore becomes more important, maybe one approach will take precedence. It's too bad Apple Silicon isn't as well documented (ie. AMX) as Intel/AMD chips are.
- tlb 5y agoIt can be a lot. Speculation requires executing operations before you're sure they'll be needed, which can double or triple power draw in order to increase instruction-level parallelism. And all the machinery needed to enable speculation, like branch predictors, draws power too. This graph shows a factor of 100 between the highest-performing and most-efficient systems: https://en.wikipedia.org/wiki/Performance_per_watt#Examples https://en.wikipedia.org/wiki/Performance_per_watt#Examples
- ece 5y agoFor general purpose compute relying more on ILP than DLP, I would think the range is going to be narrower. Narrow enough to be a meaningful choice for consumers using battery powered devices and wanting more efficient or higher performance laptops and desktops. For servers, it seems like Intel and AMD are choosing optimizing more for perf/$ over perf/watt in their roadmaps. HPC roadmaps might be different still.
- jabl 5y agoCan't say I'm terribly excited about another massive scale usage of forever chemicals, aka flourocarbons. Didn't Intel get the memo, we're trying to reduce usage of these (see e.g. the EU F-gas regulations) not increase. There's a coolant that's widely used, non-toxic, environmentally benign, cheap, abundant and non-flammable. Yeah, water. So it's not dielectric so needs some engineering. But humanity has a decent track record of building systems with pipes, hoses, heat exchangers and so forth. Same can't be said for cleaning up Superfund sites.
- antisthenes 5y ago> Same can't be said for cleaning up Superfund sites. It's a pretty huge leap to go from "closed loop CFC cooling system for a computer" to "superfund sites". What am I missing? If we're building a system with pipes and heat exchangers, why can't the coolant be a low-impact CFC rather than water? It's not a system where you just vent those cooling liquids in the atmosphere.
- BenoitP 5y agoSome of these CFC are almost eternal compounds. They are so stable that no natural light frequency from the sun can break them apart. Sulfur hexafluoride, used in high voltage circuit breakers, has a half-life of 3200 years, has a global warming potential 22800 times that of CO2. So you don't want to vent them, but any accident/leak can be considered a catastrophe. That's just the physics of it: highly dielectric + stable often makes for a big greenhouse gas offender.
- moralestapia 5y agoYup, at least Superfund is contained. This is much worse and the scale the computer consumer market is huge.
- 2muchcoffeeman 5y agoCould you use the heat? Like to pasteurise milk or something?
- shrubble 5y agoI remember reading with surprise that the Motorola 68040 CPU, which was competing with the Intel 486, could have run hotter but at a faster coock speed -- but Motorola didn't want to specify the use of a heat sink. Seems like quite a change!
- wincy 5y agoUhh, I think you made a typo that made your post quite phallic.
- aj7 5y agoTik tok is covered with these videos. I saw one with an entire server rack in a tank. https://www.tiktok.com/t/ZTdnJ8Yco/?k=1 https://www.tiktok.com/t/ZTdnJ8Yco/?k=1
- LegitShady 5y agoLTT had their mineral oil pc videos...7 years ago
- dmix 5y agoI was also exposed to this on Tiktok. The RGB trend continues, now with fish tanks...
- eternityforest 5y agoI wonder what other options there are for cooling. What's wrong with water and cooling blocks? I'm sure they could develop some quick connect hardware, paired with sensors and valves, so that any leaks could be auto-stopped. You could build the connectors such that pressing and holding the release button causes the whole loop to drain by suction, for near zero dripping as long as you wait a few seconds first.
- initplus 5y agoThere are quick release water cooling designs for the data center. But water cooling has many issues: the loop needs to be perfectly sealed as water is conductive and will instantly destroy your very expensive servers in the case of a fault. Pumps are a moving part and are prone to failure. Everything needs to be manufactured to precise tolerances. Meanwhile immersion cooling is the opposite - Just build a tank and drop your stuff in it. Modern immersion cooling fluids aren't like mineral oil so you don't end up with components permanently coated in oil.
- eternityforest 5y agoAt most, I would think it would destroy one server if there was major defect. But solenoid valves are pretty reliable. It's easy to imagine some absorbant material as a fallback, a negative pressure loop, and pressure/humidity sensors that could shut everything down within seconds of a minor leak. There's no pressure to push anything off it's connector, and in the event of anything failing and disconnecting in a major way, the loop starts draining the other way by atmospheric pressure. Water pumps are pretty cheap, you could have triple redundant pumps per rack for an irrelevant cost. Air pumps are also cheap. You could even enclose your connections to heat blocks in a second layer of slight negative pressure, that would both contain any drop-an-hour leaks that somehow happened, and funnel it to a humidity sensor. If they use anything like Fluorinert, I'm pretty sure a whole rack of this kind of tech could cost less than a single gallon of that stuff. I'm actually kind of surprised consumer water cooling setups aren't like this. It might be a pretty fun project, and you could probably use 3D printed parts depending on how much you trust the active controls.
- zmgsabst 5y agoDumb question, but why don’t we boil nitrogen for cooling? That seems to have immersion properties, without having the same toxicity.
- wmf 5y agoSub-ambient cooling has problems with condensation and thermal stress. With larger amounts of liquid nitrogen there are concerns about leaks causing asphyxiation. And of course there's the power needed to condense the nitrogen again.
- latchkey 5y agoI run a number of large datacenters. The reason why most of the worlds datacenters just have air cooling is because it is simple, easy to deploy and easy to maintain. As soon as you complicate cooling systems on a large scale, costs go up and so do failures. These are neat experiments, but you won't see it deployed in any meaningful way.
- atty 5y agoIm an outsider to data center operations, but I work in machine learning. My suspicion is that this will be driven by either the next-Gen or next-to-next-Gen GPUs/AI accelerators. It’s the curse of the field that no matter how fast the chips get, and how much memory bandwidth they get, it’s never fast enough for the state-of-the-art research. So it’s probably naturally going to require some form of liquid cooling as the chips continue to get bigger and faster. And companies that build their business around machine learning will need to adopt these deployment patterns to stay on the cutting edge.
- mirker 5y agoTPUv3 was supposed to use liquid cooling. Not sure if it actually is.
- selectodude 5y agohttps://en.wikipedia.org/wiki/Tensor_Processing_Unit#/media/File:Tensor_Processing_Unit_3.0.jpg https://en.wikipedia.org/wiki/Tensor_Processing_Unit#/media/... Sure seems to.
- nix23 5y agoYep, that's why the new "small" IBM-Mainframes are such hit (no external water-cooling anymore, just air, and the standard-rack size)
- fomine3 5y agoClickbait title?
- deleted 5y ago[deleted]
- rbanffy 5y agoIf only they could be as beautiful as the Cray 2 was (and Cray 3 would be)... As a note, I went down the single-phase immersion cooling rabbit hole (and designed a C-shaped tank for the build) but gave up when I read the instructions for disposal of the coolant (3M's Novec) and found them a little bit too complicated for my taste. I wouldn't want a fish tank of that on my living room.
- Gravityloss 5y agoThere was a research project that sent some liquid to some chip and the gates could get chemical energy from it. Then the liquid is circulated back and "recharged" and sent to the chip again. So in theory a chip could be liquid powered. The same liquid could also do cooling at the same time. Electricity by wire is of course much more convenient to use but I still wonder could there be a use cases for this. Unfortunately it's hard to find it again.