6 ms·

The hardware / datacenter side of this is equally fascinating. I used to work in AWS, but on the software / services side of things. But now and then, we would

by LASR 3y ago

The hardware / datacenter side of this is equally fascinating. I used to work in AWS, but on the software / services side of things. But now and then, we would crash some talks from the datacenter folks. One key relevation for me was that increasing compute power in DCs is primarily a thermodynamics problem than actual computing. The nodes have become so dense that shipping power in and shipping heat out, with all kinds of redundancies is an extremely hard problem. And it's not like you can perform a software update if you've discovered some inefficiencies. This was ~10 years ago, so probably some things have changed. What blows me away is that Amazon, starting out as an internet bookstore is at the cutting edge of solving thermodynamics problems.

projectileboy 3y ago
Seymour Cray used to say this all the way back in the 1970s: his biggest problems were associated with dissipating heat. For the Cray 2 he took an even more dramatic approach: "The Cray-2's unusual cooling scheme immersed dense stacks of circuit boards in a special non-conductive liquid called Fluorinert™" (https://www.computerhistory.org/revolution/supercomputers/10/68 https://www.computerhistory.org/revolution/supercomputers/10...)
- logtempo 3y ago
  Few days ago I saw an article passing by,about chips hiting the kw floor.
cogman10 3y ago
It always made me wonder why liquid cooling wasn't more of a thing for datacenters. Water has a massive amount of thermal capacity and can quickly and in bulk be cooled to optimal temperatures. You'd probably still need fans and AC to dissipate heat of non-liquid cooled parts, but for the big energy items like CPUs and GPUs/compute engines, you could ship out huge amounts of heat fairly quickly and directly. I guess the complexity and risk of a leak would be a problem, but for amazon sized data centers that doesn't seem like a major concern.
- lub 3y ago
  OVH prominently uses water cooling including custom components with their own design. https://blog.ovhcloud.com/water-cooling-from-innovation-to-disruption-part-i/ https://blog.ovhcloud.com/water-cooling-from-innovation-to-d... https://blog.ovhcloud.com/water-cooling-from-innovation-to-disruption-part-ii/ https://blog.ovhcloud.com/water-cooling-from-innovation-to-d... https://blog.ovhcloud.com/new-hybrid-immersion-liquid-cooling-developments-at-ovhcloud/ https://blog.ovhcloud.com/new-hybrid-immersion-liquid-coolin...
- adev_ 3y ago
  > It always made me wonder why liquid cooling wasn't more of a thing for datacenters. Liquid cooling is almost a defacto-standard in data centers in the HPC world. The Top of the TOP500 machines are all liquid cooled. Not by choice, but due to physics constraints. There is a big gap in power density between the HPC world and the usual datacenter-commodity-hardware world. Commodity DS are designed with the assumption that the average machine will run with a fraction of it's maximum load. HPC systems at the opposite are designed to operate safely at 100% load all the time. In a previous company where I worked, we attempted to install a medium size HPC cluster in a well-known commerical datacenter and network provider. The commercial of the DS almost felt from his chair when we announced the power requirements.
  - bayindirh 3y ago
    > we attempted to install a medium size HPC cluster in a well-known commerical Datacenter and network provider. The commercial of the DS almost fall from his chair when we announced the power requirements. Heh. We tried it too. They didn’t believe that a single node used their entire rack’s budget at first.
    - KolenCh 3y ago
      I'm going through a similar thing. Next week our procurement is arriving, but most likely majority of the nodes will sit in a box for a while before they can figure out how to power them... Also, this involves a UK university where it would takes forever to upgrade the power delivery to the building/floor/room. There's no planned upgrade whatsoever so people just need to make do with what we have.
cyrillite 3y ago
Is there any good data on the scale of this problem or that can be used to visualise it? What is the cutting edge of cooling tech like?

by LASR 3y ago

projectileboy 3y ago

Seymour Cray used to say this all the way back in the 1970s: his biggest problems were associated with dissipating heat. For the Cray 2 he took an even more dramatic approach: "The Cray-2's unusual cooling scheme immersed dense stacks of circuit boards in a special non-conductive liquid called Fluorinert™" (https://www.computerhistory.org/revolution/supercomputers/10/68 https://www.computerhistory.org/revolution/supercomputers/10...)

logtempo 3y ago
Few days ago I saw an article passing by,about chips hiting the kw floor.

cogman10 3y ago

It always made me wonder why liquid cooling wasn't more of a thing for datacenters. Water has a massive amount of thermal capacity and can quickly and in bulk be cooled to optimal temperatures. You'd probably still need fans and AC to dissipate heat of non-liquid cooled parts, but for the big energy items like CPUs and GPUs/compute engines, you could ship out huge amounts of heat fairly quickly and directly. I guess the complexity and risk of a leak would be a problem, but for amazon sized data centers that doesn't seem like a major concern.

lub 3y ago
OVH prominently uses water cooling including custom components with their own design. https://blog.ovhcloud.com/water-cooling-from-innovation-to-disruption-part-i/ https://blog.ovhcloud.com/water-cooling-from-innovation-to-d... https://blog.ovhcloud.com/water-cooling-from-innovation-to-disruption-part-ii/ https://blog.ovhcloud.com/water-cooling-from-innovation-to-d... https://blog.ovhcloud.com/new-hybrid-immersion-liquid-cooling-developments-at-ovhcloud/ https://blog.ovhcloud.com/new-hybrid-immersion-liquid-coolin...
adev_ 3y ago
> It always made me wonder why liquid cooling wasn't more of a thing for datacenters. Liquid cooling is almost a defacto-standard in data centers in the HPC world. The Top of the TOP500 machines are all liquid cooled. Not by choice, but due to physics constraints. There is a big gap in power density between the HPC world and the usual datacenter-commodity-hardware world. Commodity DS are designed with the assumption that the average machine will run with a fraction of it's maximum load. HPC systems at the opposite are designed to operate safely at 100% load all the time. In a previous company where I worked, we attempted to install a medium size HPC cluster in a well-known commerical datacenter and network provider. The commercial of the DS almost felt from his chair when we announced the power requirements.
- bayindirh 3y ago
  > we attempted to install a medium size HPC cluster in a well-known commerical Datacenter and network provider. The commercial of the DS almost fall from his chair when we announced the power requirements. Heh. We tried it too. They didn’t believe that a single node used their entire rack’s budget at first.
  - KolenCh 3y ago
    I'm going through a similar thing. Next week our procurement is arriving, but most likely majority of the nodes will sit in a box for a while before they can figure out how to power them... Also, this involves a UK university where it would takes forever to upgrade the power delivery to the building/floor/room. There's no planned upgrade whatsoever so people just need to make do with what we have.

cyrillite 3y ago

Is there any good data on the scale of this problem or that can be used to visualise it? What is the cutting edge of cooling tech like?