4 ms·

I’m a bit skeptical about the claim that the top of the TOP500 are all liquid cooled due to physical constraints. (Where the “physics” here seems to mean the de

I’m a bit skeptical about the claim that the top of the TOP500 are all liquid cooled due to physical constraints. (Where the “physics” here seems to mean the density and property of air cool in general but ignoring the environment of the machine such as the ambient weather.) The one data point I know well is NERSC, and has been air cooled until relatively recently. Part of the success of air cool in the past was the great Bay Area weather that is typically cooled enough. I forgot the exact reason for upgrading to water cool, but it was before the recent upgrade to a new machine (Perlmutter). It may have to do with our weather getting more extreme so that there are more incidences that air cool only will “throttle” your machine. The reason I’m skeptical about that claim is that the ambient temperature also play a role. But certainly the density is increasing so may be the biggest supercomputers at the moment are too dense (in power consumption) to be air cooled.

adev_ 3y ago
If you take the top5 of the top500 (01/2024): - Frontier: Liquid cooled - Aurora: liquid cooled - Eagle: liquid cooled - Fugaku: liquid cooled - Lumi: Liquid cooled with heat recycling. The need for liquid cooled arrived with the usage of accelerators that bumped significantly the power density. That said, Fugaku which is not accelerator driven is also liquid cooled today. I am surprised to learn that NERSC was air cooled until recently. Most of the BGQ machines, that were dominating the top500 10 years ago, were already liquid cooled.
- KolenCh 3y ago
  I probably wasn't clear. I don't doubt your first statement "The Top of the TOP500 machines are all liquid cooled." My skepticism is your second statement "due to physics constraints". It may very well be true but I'm just skeptical that it can't be done with a combination of engineering and ambient environment (mostly temperature but also a tradeoff between density and space). May be I focused on the word physics too much (being a physicist), but it seems the decision would basically be a cost-benefit analysis and risk management which involves many factors including money, maturity of solutions in the market, safety, etc. For example, at NERSC, there are real risk of a massive earthquake long overdue (the whole floor is quake-proof, but I guess the risk is reduced, not eliminated), so I guess they probably have considered this in that design choice. But perhaps you're right that the physics is the ultimate factor here: what comes in must goes out. With order of magnitude scale of increase in power coming in, water seems to be the most effective and cheap entity to absorb those heat and be transported outside the floor very efficiently.

adev_ 3y ago

If you take the top5 of the top500 (01/2024): - Frontier: Liquid cooled - Aurora: liquid cooled - Eagle: liquid cooled - Fugaku: liquid cooled - Lumi: Liquid cooled with heat recycling. The need for liquid cooled arrived with the usage of accelerators that bumped significantly the power density. That said, Fugaku which is not accelerator driven is also liquid cooled today. I am surprised to learn that NERSC was air cooled until recently. Most of the BGQ machines, that were dominating the top500 10 years ago, were already liquid cooled.

KolenCh 3y ago
I probably wasn't clear. I don't doubt your first statement "The Top of the TOP500 machines are all liquid cooled." My skepticism is your second statement "due to physics constraints". It may very well be true but I'm just skeptical that it can't be done with a combination of engineering and ambient environment (mostly temperature but also a tradeoff between density and space). May be I focused on the word physics too much (being a physicist), but it seems the decision would basically be a cost-benefit analysis and risk management which involves many factors including money, maturity of solutions in the market, safety, etc. For example, at NERSC, there are real risk of a massive earthquake long overdue (the whole floor is quake-proof, but I guess the risk is reduced, not eliminated), so I guess they probably have considered this in that design choice. But perhaps you're right that the physics is the ultimate factor here: what comes in must goes out. With order of magnitude scale of increase in power coming in, water seems to be the most effective and cheap entity to absorb those heat and be transported outside the floor very efficiently.