4 ms·

Cool stuff. I remember seeing liquid cooling solutions that immersed the entire system in some non-conductive fluid, like mineral oil. Of course, that's a bit m

by titzer 5y ago

Cool stuff. I remember seeing liquid cooling solutions that immersed the entire system in some non-conductive fluid, like mineral oil. Of course, that's a bit messy, but I wonder if all components were designed to be immersed, whether it'd be feasible to consider the entire server enclosure as the watertight unit, e.g. in a rack, rather than hoses to individual servers.

toast0 5y ago
Immersing the rack as a unit means commiting to servicing the rack as a unit. Some organizations do that. Bring the rack up, do a burn in test, then run it until enough bits are broken that it makes sense to take the whole thing down to swap bits or until it's obsolete enough to justify replacing the whole thing. But a lot of organizations want to replace storage or ram or ?? as it fails without turning off the whole rack. Would bring more meaning to the words 'draining' a server/rack/data center though. Actually reading the article, they have immersion chambers which look more or less like a rack rotated 90 to be horizontal. Which means you can pull a rack mount server out vertically to service it, but it also takes up a lot more floorspace (at least 3x to my eye), which reduces density. If you were OK with reduced density, you could leave your racks 2/3rds empty or use 2U servers instead of 1U servers, etc. Power and cooling requirements are already the bottleneck for most datacenters rather than floorspace, and liquid cooling doesn't really help a lot there (unless you can pump your working fluid outdoors, but usually it's just dumping the heat elsewhere in the room more effectively, sometimes with more power for a pump than for fans).
- elihu 5y ago
  I once saw a setup with a whole rack laid down on its back in a tub of mineral oil, like you describe. You could slide individual servers up and set them on the top of the enclosure, letting them drip down into the tub. Kind of messy, but not unreasonably so. (This was setup at SC'09 in Portland. I'm impressed that someone went to the trouble to set this thing up just for a few days of a conference.) The oil was actively pumped through a radiator with a fan. Part of the whole point of the system is that, since oil is a better coolant than air, they can run the whole system at a higher ambient temperature, like 80 or 90 degrees F instead of 70-ish. That could reduce energy costs quite a bit. I'm not sure if the oil goes bad eventually and has to be replaced; I could see that being expensive if there isn't a simple way to clean/recycle/reprocess it back into usable oil.
- fomine3 5y ago
  In other words, HPC
PragmaticPulp 5y ago
Submersion cooling systems are shown on page 3 of the article. The page selector is at the bottom of the text. I like how they have to have hoists to raise and lower the servers into the bath because they're so heavy.
- titzer 5y ago
  Cool, I should have read more of the article.

by titzer 5y ago

toast0 5y ago

Immersing the rack as a unit means commiting to servicing the rack as a unit. Some organizations do that. Bring the rack up, do a burn in test, then run it until enough bits are broken that it makes sense to take the whole thing down to swap bits or until it's obsolete enough to justify replacing the whole thing. But a lot of organizations want to replace storage or ram or ?? as it fails without turning off the whole rack. Would bring more meaning to the words 'draining' a server/rack/data center though. Actually reading the article, they have immersion chambers which look more or less like a rack rotated 90 to be horizontal. Which means you can pull a rack mount server out vertically to service it, but it also takes up a lot more floorspace (at least 3x to my eye), which reduces density. If you were OK with reduced density, you could leave your racks 2/3rds empty or use 2U servers instead of 1U servers, etc. Power and cooling requirements are already the bottleneck for most datacenters rather than floorspace, and liquid cooling doesn't really help a lot there (unless you can pump your working fluid outdoors, but usually it's just dumping the heat elsewhere in the room more effectively, sometimes with more power for a pump than for fans).

elihu 5y ago
I once saw a setup with a whole rack laid down on its back in a tub of mineral oil, like you describe. You could slide individual servers up and set them on the top of the enclosure, letting them drip down into the tub. Kind of messy, but not unreasonably so. (This was setup at SC'09 in Portland. I'm impressed that someone went to the trouble to set this thing up just for a few days of a conference.) The oil was actively pumped through a radiator with a fan. Part of the whole point of the system is that, since oil is a better coolant than air, they can run the whole system at a higher ambient temperature, like 80 or 90 degrees F instead of 70-ish. That could reduce energy costs quite a bit. I'm not sure if the oil goes bad eventually and has to be replaced; I could see that being expensive if there isn't a simple way to clean/recycle/reprocess it back into usable oil.
fomine3 5y ago
In other words, HPC

PragmaticPulp 5y ago

Submersion cooling systems are shown on page 3 of the article. The page selector is at the bottom of the text. I like how they have to have hoists to raise and lower the servers into the bath because they're so heavy.

titzer 5y ago
Cool, I should have read more of the article.