3 ms·
> I can't even imagine the power draw of such a system or the heat generation. I’m unfamiliar with the machine specified in your example, but I’ve seen in pers
by jsjohnst 5y ago
> I can't even imagine the power draw of such a system or the heat generation.
I’m unfamiliar with the machine specified in your example, but I’ve seen in person HPC clusters with more RAM than that, so I’ll use that to give you some napkin math calculations.
Dell R730 with 24x64gb ECC DIMMs = 1536gb RAM. To get to 256TB, you’d need about 170 of them (aka about 5 racks). Power draw varies depending on a number of factors, but let’s be very conservative and take a worse case and say 750W per 1U server. That’s 127,500W, which is just over 600A at 208V, so say a long block of houses with their A/Cs cranking in the summer. Now as to the heat generation, each R730 puts out up to 2850 BTU/hr, so for all 170 of them, that’s just shy of 500,000 BTU/hr. Continuing the comparison to homes, that’s 5-6 home furnaces worth of heat generation per hour.
I don’t have any photos handy of the 64gb DIMMs referenced above, but here’s a photo of 130 16gb ECC DIMMs (aka just over 2TB):
https://www.dropbox.com/s/ww1fwdn7o9se14z/2020-06-15%2020.39.31.jpg?dl=0 https://www.dropbox.com/s/ww1fwdn7o9se14z/2020-06-15%2020.39...
If instead of the above based on a real world HPC machine, you based it on current gen Dell R750s which can hold 32 256gb LRDIMMs totaling 8TB per machine, you could get to by with just over a single rack of machines for 256TB, albeit for a much higher price (256gb LRDIMMs cost as much each as roughly the price of a used R730 complete w/ 1.5TB of RAM). The cluster of R750s with that much RAM configured as indicated would be well into the 8 figure range.