4 ms·

Oh, and not to tar with toooo broad a brush: One guy contacted me the other day wanting a terabyte of memory. He'd actually done all his homework and had a weir

Oh, and not to tar with toooo broad a brush: One guy contacted me the other day wanting a terabyte of memory. He'd actually done all his homework and had a weirdly interesting use case. I'm actually pretty excited to explore getting his problem solved; it's likely that a terabyte is about the right amount to get him over the finish line, and he's converged on this algorithm precisely because it has fairly horrifying memory needs, but trades off to give a nice big-O runtime.

arthurcolle 5y ago
I've heard that some financial institutions use a YARC supercomputer with 256TB of RAM to do daily "full market watch" of client activity to determine insider trading for compliance purposes. I can't even imagine the power draw of such a system or the heat generation. Fucking galaxy brain engineering. At that point you need like custom motherboards and special buses, right?
- chunkyks 5y ago
  Finance guys tend to be able to afford to have the right people to be sure of that on hand; they also are specifically in the business of cost-benefit analysis. I have no insight or information this; but sometimes you can throw money at a problem to make it go away, and that's literally the best deal open to you. If custom fab solves a problem more cheaply than the 25 senior software engineers it would otherwise take, then custom fab it is.
- pinewurst 5y ago
  That’s about a 2,000 node Cray, something like 14 cabinets which isn’t that big of an XC configuration.
- jsjohnst 5y ago
  > I can't even imagine the power draw of such a system or the heat generation. I’m unfamiliar with the machine specified in your example, but I’ve seen in person HPC clusters with more RAM than that, so I’ll use that to give you some napkin math calculations. Dell R730 with 24x64gb ECC DIMMs = 1536gb RAM. To get to 256TB, you’d need about 170 of them (aka about 5 racks). Power draw varies depending on a number of factors, but let’s be very conservative and take a worse case and say 750W per 1U server. That’s 127,500W, which is just over 600A at 208V, so say a long block of houses with their A/Cs cranking in the summer. Now as to the heat generation, each R730 puts out up to 2850 BTU/hr, so for all 170 of them, that’s just shy of 500,000 BTU/hr. Continuing the comparison to homes, that’s 5-6 home furnaces worth of heat generation per hour. I don’t have any photos handy of the 64gb DIMMs referenced above, but here’s a photo of 130 16gb ECC DIMMs (aka just over 2TB): https://www.dropbox.com/s/ww1fwdn7o9se14z/2020-06-15%2020.39.31.jpg?dl=0 https://www.dropbox.com/s/ww1fwdn7o9se14z/2020-06-15%2020.39... If instead of the above based on a real world HPC machine, you based it on current gen Dell R750s which can hold 32 256gb LRDIMMs totaling 8TB per machine, you could get to by with just over a single rack of machines for 256TB, albeit for a much higher price (256gb LRDIMMs cost as much each as roughly the price of a used R730 complete w/ 1.5TB of RAM). The cluster of R750s with that much RAM configured as indicated would be well into the 8 figure range.
- rachelbythebay 5y ago
  If you're allowed to call a multi-node setup a "supercomputer", then how about 3 40 machine racks of Xeons where each one is dual-socketed, has 12 DIMMs per socket, and is using 128 GB DIMMs? 3 racks * 40 boxes * 2 sockets * 12 DIMMs per socket * 128 GB = 368640 GB or 360 TB raw. Remove some of that for overhead (kernel, OS gunk, network buffers, whatever code you'd have to run to make it into a cluster), and there's your 256 TB with a bunch left over. Stuff like this is why I sometimes say that a lot of these startups could be done with "three Xeon racks" if they actually cared about doing it right. [Note: assumes 2nd generation Xeons (Skylakes), with 24 DIMMs. Ice Lake = more/bigger memory, fewer actual machines.]

arthurcolle 5y ago

I've heard that some financial institutions use a YARC supercomputer with 256TB of RAM to do daily "full market watch" of client activity to determine insider trading for compliance purposes. I can't even imagine the power draw of such a system or the heat generation. Fucking galaxy brain engineering. At that point you need like custom motherboards and special buses, right?

chunkyks 5y ago
Finance guys tend to be able to afford to have the right people to be sure of that on hand; they also are specifically in the business of cost-benefit analysis. I have no insight or information this; but sometimes you can throw money at a problem to make it go away, and that's literally the best deal open to you. If custom fab solves a problem more cheaply than the 25 senior software engineers it would otherwise take, then custom fab it is.
pinewurst 5y ago
That’s about a 2,000 node Cray, something like 14 cabinets which isn’t that big of an XC configuration.
jsjohnst 5y ago
> I can't even imagine the power draw of such a system or the heat generation. I’m unfamiliar with the machine specified in your example, but I’ve seen in person HPC clusters with more RAM than that, so I’ll use that to give you some napkin math calculations. Dell R730 with 24x64gb ECC DIMMs = 1536gb RAM. To get to 256TB, you’d need about 170 of them (aka about 5 racks). Power draw varies depending on a number of factors, but let’s be very conservative and take a worse case and say 750W per 1U server. That’s 127,500W, which is just over 600A at 208V, so say a long block of houses with their A/Cs cranking in the summer. Now as to the heat generation, each R730 puts out up to 2850 BTU/hr, so for all 170 of them, that’s just shy of 500,000 BTU/hr. Continuing the comparison to homes, that’s 5-6 home furnaces worth of heat generation per hour. I don’t have any photos handy of the 64gb DIMMs referenced above, but here’s a photo of 130 16gb ECC DIMMs (aka just over 2TB): https://www.dropbox.com/s/ww1fwdn7o9se14z/2020-06-15%2020.39.31.jpg?dl=0 https://www.dropbox.com/s/ww1fwdn7o9se14z/2020-06-15%2020.39... If instead of the above based on a real world HPC machine, you based it on current gen Dell R750s which can hold 32 256gb LRDIMMs totaling 8TB per machine, you could get to by with just over a single rack of machines for 256TB, albeit for a much higher price (256gb LRDIMMs cost as much each as roughly the price of a used R730 complete w/ 1.5TB of RAM). The cluster of R750s with that much RAM configured as indicated would be well into the 8 figure range.
rachelbythebay 5y ago
If you're allowed to call a multi-node setup a "supercomputer", then how about 3 40 machine racks of Xeons where each one is dual-socketed, has 12 DIMMs per socket, and is using 128 GB DIMMs? 3 racks * 40 boxes * 2 sockets * 12 DIMMs per socket * 128 GB = 368640 GB or 360 TB raw. Remove some of that for overhead (kernel, OS gunk, network buffers, whatever code you'd have to run to make it into a cluster), and there's your 256 TB with a bunch left over. Stuff like this is why I sometimes say that a lot of these startups could be done with "three Xeon racks" if they actually cared about doing it right. [Note: assumes 2nd generation Xeons (Skylakes), with 24 DIMMs. Ice Lake = more/bigger memory, fewer actual machines.]