6 ms·

You can get commodity x86 server with 12TB RAM and 224 cores.

by qaq 8y ago

toast0 8y ago
It looks like 12TB ram is available in eight socket Intel servers? Those are sort of commodity, but availability is limited, and NUMA becomes a much larger issue than in the more easily available dual socket configurations. Looks like Epyc can do 4TB in a dual socket configuration.
- deleted 8y ago
  [deleted]
paulddraper 8y ago
Link? It depends what is meant by "commodity" I guess. The largest server AWS offers is only 64 physical cores (128 logical) and less than 4 TB RAM.
- qaq 8y ago
  https://lenovopress.com/lp0647-thinksystem-sr950-server https://lenovopress.com/lp0647-thinksystem-sr950-server
  - paulddraper 8y ago
    Interesting, though I don't see the price or any other purchasing details (at least on that page).
    - jandrese 8y ago
      Pretty sure once you're talking about 12TB of memory it goes to "call for pricing".
      - qaq 8y ago
        I am guessing that it would be in 250-270K range
      - paulddraper 8y ago
        That seems more like enterprise built and sold specially, and less like commodity hardware.
        qaq 8y ago
        To me things available from grey box x86 vendors like supermicro or lenovo look like commodity hardware.
    - travisoneill1 8y ago
      51K CAD https://www.frontierpc.com/computers/entry-level-servers/server/lenovo/thinksystem-sr950/7x12a01mna-server-1040955612.html https://www.frontierpc.com/computers/entry-level-servers/ser...
    - qaq 8y ago
      here's online configurator for Supermicro 8-way box https://www.rackmountpro.com/product/2570/7089P-TR4T.html https://www.rackmountpro.com/product/2570/7089P-TR4T.html
- chx 8y ago
  http://www.supermicro.com/products/system/4U/8048/SYS-8048B-TR4FT.cfm http://www.supermicro.com/products/system/4U/8048/SYS-8048B-... Quad socket R1 (LGA 2011) supports Intel® Xeon® processor E7-8800 v4/v3, E7-4800 v4/v3 family (up to 24-Core) Up to 12TB DDR4 (128GB 3DS LRDIMM); 96x DIMM slots (8x memory module boards: X10QBi-MEM2) AFAIK the latest Skylake Xeons ("Scalable" - Platinum/Gold/Silver/Bronze) have regressed to 1.5TB support, see https://ark.intel.com/products/93794/Intel-Xeon-Processor-E7-4809-v4-20M-Cache-2_10-GHz https://ark.intel.com/products/93794/Intel-Xeon-Processor-E7... http://www.colfax-intl.com/nd/downloads/Intel-Xeon-Scalable-Processor-Matrix.pdf http://www.colfax-intl.com/nd/downloads/Intel-Xeon-Scalable-... corroborated by https://ark.intel.com/products/120502/Intel-Xeon-Platinum-8160M-Processor-33M-Cache-2_10-GHz https://ark.intel.com/products/120502/Intel-Xeon-Platinum-81... Since a single 128GB stick costs about 2900 USD, 96 of them will run to ~280 000 USD plus the server so it's likely to be above 300 000K. If you want to go with 6TB "only" then it's a lot, lot cheaper as 64GB RDIMM sticks can be had below 700 USD. The end result might cost closer to a third of the 12TB server than half of it. RAM prices comparison: https://memory.net/memory-prices/ https://memory.net/memory-prices/
  - mmt 8y ago
    That's a previous-gen CPU system. Current is: https://www.supermicro.com/products/system/7U/7089/SYS-7089P-TR4T.cfm https://www.supermicro.com/products/system/7U/7089/SYS-7089P... Server itself is might only be $25k but those 224 cores could add another $90k, so the total would be close to $400k. The previous-gen version, SYS-7088B-TR4FT (link in my comment upthread), has 192 DIMM slots, so if you don't need CPU horsepower, you cn get the lower-density modules and still have 12TB (or the max of 24TB for the price premium!).
    - chx 8y ago
      I edited my answer to indicate that current gen 3.06TB RAM supporting CPUs are not out (yet?)
      - mmt 8y ago
        Right, which is why the current generation is limited to 12TiB for 8S systems, compared to 24TiB for the generation you quoted. Even previous gen, if 12TiB main memory is your goal (NUMA concerns aside), it's probably worth going for the 8S system instead of the 4S one, since that's a savings of $144k, and 8 slower/fewer-core CPUs might even be cheaper than 4 that have twice the performance.
mmt 8y ago
If you're willing to go back a generation on CPU (and merely 192 cores), you can get 24TB. https://www.supermicro.com/products/system/7U/7088/SYS-7088B-TR4FT.cfm https://www.supermicro.com/products/system/7U/7088/SYS-7088B...
e12e 8y ago
Anyone have experience with mainframes when it comes to this level of cost/performance? I suppose 500k USD is still too "cheap" to consider more special hw/systems - but where's the cut-off when getting a monster running db2 and Code in a Linux VM or something?

You can get commodity x86 server with 12TB RAM and 224 cores.

by qaq 8y ago

You can get commodity x86 server with 12TB RAM and 224 cores.

It looks like 12TB ram is available in eight socket Intel servers? Those are sort of commodity, but availability is limited, and NUMA becomes a much larger issue than in the more easily available dual socket configurations. Looks like Epyc can do 4TB in a dual socket configuration.

deleted 8y ago
[deleted]

paulddraper 8y ago

Link? It depends what is meant by "commodity" I guess. The largest server AWS offers is only 64 physical cores (128 logical) and less than 4 TB RAM.

qaq 8y ago
https://lenovopress.com/lp0647-thinksystem-sr950-server https://lenovopress.com/lp0647-thinksystem-sr950-server
- paulddraper 8y ago
  Interesting, though I don't see the price or any other purchasing details (at least on that page).
  - jandrese 8y ago
    Pretty sure once you're talking about 12TB of memory it goes to "call for pricing".
    - qaq 8y ago
      I am guessing that it would be in 250-270K range
    - paulddraper 8y ago
      That seems more like enterprise built and sold specially, and less like commodity hardware.
      - qaq 8y ago
        To me things available from grey box x86 vendors like supermicro or lenovo look like commodity hardware.
  - travisoneill1 8y ago
    51K CAD https://www.frontierpc.com/computers/entry-level-servers/server/lenovo/thinksystem-sr950/7x12a01mna-server-1040955612.html https://www.frontierpc.com/computers/entry-level-servers/ser...
  - qaq 8y ago
    here's online configurator for Supermicro 8-way box https://www.rackmountpro.com/product/2570/7089P-TR4T.html https://www.rackmountpro.com/product/2570/7089P-TR4T.html
chx 8y ago
http://www.supermicro.com/products/system/4U/8048/SYS-8048B-TR4FT.cfm http://www.supermicro.com/products/system/4U/8048/SYS-8048B-... Quad socket R1 (LGA 2011) supports Intel® Xeon® processor E7-8800 v4/v3, E7-4800 v4/v3 family (up to 24-Core) Up to 12TB DDR4 (128GB 3DS LRDIMM); 96x DIMM slots (8x memory module boards: X10QBi-MEM2) AFAIK the latest Skylake Xeons ("Scalable" - Platinum/Gold/Silver/Bronze) have regressed to 1.5TB support, see https://ark.intel.com/products/93794/Intel-Xeon-Processor-E7-4809-v4-20M-Cache-2_10-GHz https://ark.intel.com/products/93794/Intel-Xeon-Processor-E7... http://www.colfax-intl.com/nd/downloads/Intel-Xeon-Scalable-Processor-Matrix.pdf http://www.colfax-intl.com/nd/downloads/Intel-Xeon-Scalable-... corroborated by https://ark.intel.com/products/120502/Intel-Xeon-Platinum-8160M-Processor-33M-Cache-2_10-GHz https://ark.intel.com/products/120502/Intel-Xeon-Platinum-81... Since a single 128GB stick costs about 2900 USD, 96 of them will run to ~280 000 USD plus the server so it's likely to be above 300 000K. If you want to go with 6TB "only" then it's a lot, lot cheaper as 64GB RDIMM sticks can be had below 700 USD. The end result might cost closer to a third of the 12TB server than half of it. RAM prices comparison: https://memory.net/memory-prices/ https://memory.net/memory-prices/
- mmt 8y ago
  That's a previous-gen CPU system. Current is: https://www.supermicro.com/products/system/7U/7089/SYS-7089P-TR4T.cfm https://www.supermicro.com/products/system/7U/7089/SYS-7089P... Server itself is might only be $25k but those 224 cores could add another $90k, so the total would be close to $400k. The previous-gen version, SYS-7088B-TR4FT (link in my comment upthread), has 192 DIMM slots, so if you don't need CPU horsepower, you cn get the lower-density modules and still have 12TB (or the max of 24TB for the price premium!).
  - chx 8y ago
    I edited my answer to indicate that current gen 3.06TB RAM supporting CPUs are not out (yet?)
    - mmt 8y ago
      Right, which is why the current generation is limited to 12TiB for 8S systems, compared to 24TiB for the generation you quoted. Even previous gen, if 12TiB main memory is your goal (NUMA concerns aside), it's probably worth going for the 8S system instead of the 4S one, since that's a savings of $144k, and 8 slower/fewer-core CPUs might even be cheaper than 4 that have twice the performance.

mmt 8y ago

If you're willing to go back a generation on CPU (and merely 192 cores), you can get 24TB. https://www.supermicro.com/products/system/7U/7088/SYS-7088B-TR4FT.cfm https://www.supermicro.com/products/system/7U/7088/SYS-7088B...

e12e 8y ago

Anyone have experience with mainframes when it comes to this level of cost/performance? I suppose 500k USD is still too "cheap" to consider more special hw/systems - but where's the cut-off when getting a monster running db2 and Code in a Linux VM or something?