6 ms·
Servers Too Hot? Intel Recommends a Luxurious Oil Bath
- amalag 14y agoI really cannot understand why this is not more prevalent. The cost savings of not having air conditioning for a datacenter should be well worth any mess of oil.
- amalag 14y agoThe articles around this are interesting and consider how it could become prevalent and they rightfully point to high performance computing. I think another obvious area is countries like India which have very high costs for power and do not have much in the way of existing datacenters.
- deleted 14y ago[deleted]
- tmurray 14y agoImmersion cooling was used on the Cray-2 supercomputer with Fluorinert[1]. Water cooling is fairly common in high-end supercomputing setups (I think all the BlueGene systems are water-cooled, for example). [1] http://en.wikipedia.org/wiki/Cray-2 http://en.wikipedia.org/wiki/Cray-2
- hristov 14y agoIt is not prevalent because they are still testing it and the oil servers are not yet available for purchase. I do hope when they do become available that the price is not too high.
- a5seo 14y agoseems close: http://www.asetek.com/server/sealed-server.html http://www.asetek.com/server/sealed-server.html
- deleted 14y ago[deleted]
- ars 14y agoThere are lots of problems. First the standard vertical rack wouldn't work - you would not be able to remove any blades from the rack since they are submerged. You would not be able to lift the entire rack out of the tub, since you might not have enough height, but worse the other computers in that rack would probably burn up from a sudden loss of cooling. So you would have to switch to horizontal racks, so you would lift one blade out without affecting the others. Of course that wastes tons of space. If you want lots of servers you would need multiple racks one above the other - but then how would you service the upper ones? And you are wasting space again - you need air height above each rack. Normally in a vertical server, the gap you need to remove the server is the same gap you need to walk there, so no extra wasted space. And then there are weight issues - a typical 21.5 inch x 19 inch x 8 feet rack would weigh about half a ton just from mineral oil! (Not including any metal.) Good luck building something strong enough to hold lots of them, and forget mounting them one above the other. All of this can be designed around, but it's not a simple matter - for now you would need to make it custom.
- ksec 14y agoMy thoughts exactly, although most of the time Space should be cheap enough where the DC are located. But you would need at least 2 - 3x the size of DC to handle that. The Oil also needs money, and the extra cost to handle the construction, etc. And if google could already achieve 10- 20% figure then it is properly not worth while to do it within that 18% savings.
- PayUpPal 14y agoBlades in huge datacenters are not serviced one by one. Typically you'd wait until say 20% of the blades fail and only then fix and upgrade the whole rack at once.
- grandalf 14y agoWhat you describe are not faults they are tradeoffs. Perhaps instead of having 14' ceilings and sub-floors and AC a datacenter would have 6' ceilings and no AC. Whether the technology makes sense depends on the overall cost effectiveness. Google putting a data center near a river and using the river water in a complex water cooling system, or routing used non-potable water to provide cooling are both systems that are used today and likely have at least as many challenging aspects, yet are in use today b/c they lower overall costs.
- mistercow 14y agoIt's fascinating that I heard about homebrewers doing this kind of thing around a decade ago (although I believe they were keeping the hard drives un-submerged rather than sealing them up) yet this is only now being considered "cutting edge" as a commercial option.
- yungchin 14y agoYup, I was reminded of those too! Feeding the nostalgia: http://web.archive.org/web/20010623203101/http://www.hardwareoc.com/submersed.php http://web.archive.org/web/20010623203101/http://www.hardwar...
- mistercow 14y agoI like how the author of that completely did not understand how heat conduction works: Just submersing your components in a liquid doesn't do you much good as the liquid will be room temperature and your CPU will burn up just as fast as if it were in the open. Well no, because see, the thermal conductivity of air is about 0.026 W/mK and the thermal conductivity of mineral oil is at least 0.1 W/mK and oh god I'm trying to argue with someone 11 years in the past.
- sliverstorm 14y agoIt's just a reminder that "commercially viable" is not the same thing as "possible".
- mistercow 14y agoYet I wonder what really has changed that makes it commercially viable now. Pumping fluids around isn't a technology that has significantly advanced, after all. Is it just that Moore's law has ceased to apply to heat and power usage? Is it that CPU manufacturers finally got around to testing and falsifying the long-assumed-true hypothesis that servers need to be kept in ultra cool rooms to begin with?
- sliverstorm 14y ago
- w1ntermute 14y agoReminds me of a Bioshock Casemod from a few years ago[0]. From what the creator said, it doesn't even sound that expensive. 0: http://www.reddit.com/r/gaming/comments/dmrpv/bioshock_mineral_oil_media_center_case/ http://www.reddit.com/r/gaming/comments/dmrpv/bioshock_miner...
- evoxed 14y agoInteresting that supermicro is already moving towards putting these on the market. I hadn't heard of them until the other day with the etsy setup, but this looks very promising for future datacenters large and small...
- webwielder 14y agoMidas Green Tech, anyone? http://www.midasgreentech.com http://www.midasgreentech.com
- achristensen 14y agoCheck out LiquidCool Solutions too. I think this type of technology could very well be the standard in the future. http://www.liquidcoolsolutions.com http://www.liquidcoolsolutions.com
- elorant 14y agoI recently saw a video about a similar implementation at an annual conference in Germany for PC modders that won first place. It looked pretty awesome, the whole PC box was immersed in a greenish liquid. You can see it here, it's at the end of the video after 8:33. http://www.youtube.com/watch?v=HMduN09xCgs&feature=player_embedded http://www.youtube.com/watch?v=HMduN09xCgs&feature=playe...
- cnvogel 14y agoThis of course looks very cool, but to me it does not seem practical besides edge-cases, even though the technology is not new in any way, high-powered transformers which you find in power plants, for example, are oil-filled since basically forever. The practical problems on the other hand are not to be ignored: The oil needs to be kept rather clean, otherwise it will loose its good insulation properties, hence you need sealed containers or purification devices, in industrial scale deployments this means keeping an eye on the chemical composition of your coolant by regular chemical analysis. Connectors and cables need to be really oil-tight, otherwise the oil will creep out through cables hanging out of a closed vessel (even if they go higher than the oil-surface)! It's not an efficient technology in terms of the amount of coolant that is used: You put everything into the oil-bath, both the high-energy-density components (CPU, graphics-card, maybe chipsets, fast RAM), regardless of their energy consumption: Imagine a full data center, you'd basically need thousands of cubic meters of pure, constantly filtered oil... For the good old cray-computer often used as an example it was different: There computing was spread out over a vast number of logic gates that deposited their waste heat over a really high volume. The examples I have seen (casemodders being a good example) were also rather generous with container-size and amount of coolant where enough circulation and cooling was provided by natural convection. "Professionally", one would use this technology for a reason, hence in devices that are even more densly packed than usual blade-centers and there I expect issues with forced circulations of warm/hot oil just as we have hot/cold air distribution nowadays. A technology like currently deployed water-cooled devices, where the majority of heat is collected by water at the concentration points (mainly CPU) and air is taking care of the (much reduced) remains seems much more sensible to me.
- sigkill 14y agoThis may be only partially relevant, but I'm reading about transformer protectors and I can see all sorts of things going wrong here. Extreme heat causes arcing in the oil and consequently fire and/or explosion, in transformers. I don't know how that'd work here.
- mistercow 14y agoArcing is a problem with open air transformers too, and one of the functions of oil in oil-filled transformers is to prevent arcing. Probably this property simply breaks down at very high temperatures, but since arcing isn't generally a problem for computers in the first place, it shouldn't be a problem regardless of oil temperature.
- gary4gar 14y agoNo risk of short-circuits?
- raverbashing 14y agoNo, mineral oil as they're using it, with required specs, is an even better insulator than air
- rwmj 14y agoAccording to the article, Intel recommends removing the grease between the processor and the heat sink. Question: Why does it need a heat sink if it's being cooled by oil?
- raverbashing 14y agoWell, the obvious answer is that heat resistance for that processor area to oil is insufficient to remove all the heat Hence you still need a heatsink (smaller though and no need for a fan)
- Tomdarkness 14y agoLarger surface area to dissipate the heat from the CPU to the oil?
- lutusp 14y ago> Why does it need a heat sink if it's being cooled by oil? The heatsink couples the heat source (the electronics) to the oil. The metal heatsink has much higher thermal conductivity than the oil. The advantage of the oil is that it circulates and carries heat away to another heatsink that ultimately dissipates heat to the air.
- xymostech 14y agoI interpreted that more as "After you remove the heat sink, make sure there isn't any of the conducting grease left over because it can ruin the mineral oil" but they could mean using a heat sink with no grease as well.
- rdwallis 14y agoI built a mineral oil PC a couple of years ago. I epoxied the motherboard mount from an old computer case to some wood and then stuck the wood to the inside of a cheap fish tank. I installed the computer sans harddrive and then poured in 12 liters of mineral oil that I'd bought from a vet (Vet's use it as a horse laxative). It resulted in a perfectly silent system but because I didn't cover the top it was also a really effective fly trap. Within a few months there were a bunch of dead flies floating at the top of the tank. Also the weight made it a real effort to move around and no matter how careful I was there was always a mess when I needed to swap out or adjust a part. Still it was a lot of fun to build and I used it as my primary machine for about a year.
- mebesilly 14y agoMineral oil is a Group 1 carcinogen. Scope for some lucrative class-action litigation 10 years down the line when workers in data centres start developing carcinomas.