4 ms·

The processors they're using appear to be well in excess of $10,000 each. To get 224 physical cores, you'd need to run them in a 8-CPU configuration, so you've

The processors they're using appear to be well in excess of $10,000 each. To get 224 physical cores, you'd need to run them in a 8-CPU configuration, so you've got ~$100,000 invested just in the processors alone. Memory is absurdly expensive right now, and I'd be shocked if 12TB of it cost any less than another $100,000. You have to sign a multi-year reservation, and with a cost of $267,512.88 per year (as calculated from another comment in the thread), I assume their profit margins for the first year are nearly non-existent. However, over the course of the remaining two years of your reservation, they're making a great deal of money on each reservation. So yea, I imagine someone could build something like this, but such an individual would need to have very deep pockets.

slededit 8y ago
12TB is only about $7000 in chips. I buy them for my own products. There’s a lot of ancillary things you need but raw dimms aren’t anywhere near that experience.
- maximilianburke 8y ago
  The 128GB DIMMs are nearly $3000 each (https://memory.net/product/m386aak40b40-cwd-samsung-1x-128gb-ddr4-2666-lrdimm-pc4-21300v-l-octa-rank-x4-module/ https://memory.net/product/m386aak40b40-cwd-samsung-1x-128gb...) and you need 96 of them. Where can you get 12tb for $7000?
  - deleted 8y ago
    [deleted]
  - slededit 8y ago
    You'll pay more for the highest density chips, but there are back-planes that support more of the lower density parts. I buy the raw chips in reels not DIMMs. The $7k was based on my costs.
- spydum 8y ago
  ECC DIMMS at the density required for this probably going to run you substantially more.List price for an ECC DDR4 128GB is like close to $10K US each Nobody pays list, but still.. I’m guessing they have something like 48x256gb dimms — that is serious coin.
etaioinshrdlu 8y ago
Can one even buy an 8 socket motherboard off the shelf?
- Rychard 8y ago
  Once you go beyond dual socket boards (and even then, they're not widely available), you're firmly in the territory of custom form factors, and extensibility usually happens via proprietary daughterboards. At this point it's not really consumer-grade hardware though, so I wouldn't really classify it under "off the shelf".
  - mmt 8y ago
    The "custom" and "proprietary" are largely irrelevant, in that they're merely internal implementation details. These systems (from SuperMicro or Lenovo, for example) still fit in a standard 19" rack, and still take standard CPUs, standard DIMMs, and standard PCIe cards. The main limitation is that the complete "barebones" system is indivisible, unlike with most 1S-2S systems, where motherboards can be mixed and matched among a wider variety of chasses. This limitation also applies to high-density offerings like blades and SM's "Twin" line. I have yet to encounter a situation where such a limitation has ever even entered the conversation.
    - Rychard 8y ago
      > The "custom" and "proprietary" are largely irrelevant, in that they're merely internal implementation details. Agreed. I was merely trying to convey that they wouldn't be the normal [E]ATX form factor boards. The parent was asking for something "off the shelf", and I took this to mean purchasing a motherboard by itself. You could get something like that from SM[0], but I doubt you'd be able to find a compatible chassis for the motherboard from anyone other than SM. At that point it probably makes more sense to just buy the system[1]. [0] http://www.supermicro.com/products/motherboard/Xeon/C600/X10QRH_.cfm http://www.supermicro.com/products/motherboard/Xeon/C600/X10... [1] http://www.supermicro.com/products/system/2U/2048/SYS-2048U-RTR4.cfm http://www.supermicro.com/products/system/2U/2048/SYS-2048U-...
mmt 8y ago
> cost of $267,512.88 per year (as calculated from another comment in the thread), I assume their profit margins for the first year are nearly non-existent NB that this was calculated based on the "All Up Front" pricing, so, for that situation, it's safer to say that their profit for the first year is $300k+ [1] and merely non-existent or negative for the second and third years. [1] Assuming $400k purchase cost as per https://news.ycombinator.com/item?id=18090058 https://news.ycombinator.com/item?id=18090058

slededit 8y ago

12TB is only about $7000 in chips. I buy them for my own products. There’s a lot of ancillary things you need but raw dimms aren’t anywhere near that experience.

maximilianburke 8y ago
The 128GB DIMMs are nearly $3000 each (https://memory.net/product/m386aak40b40-cwd-samsung-1x-128gb-ddr4-2666-lrdimm-pc4-21300v-l-octa-rank-x4-module/ https://memory.net/product/m386aak40b40-cwd-samsung-1x-128gb...) and you need 96 of them. Where can you get 12tb for $7000?
- deleted 8y ago
  [deleted]
- slededit 8y ago
  You'll pay more for the highest density chips, but there are back-planes that support more of the lower density parts. I buy the raw chips in reels not DIMMs. The $7k was based on my costs.
spydum 8y ago
ECC DIMMS at the density required for this probably going to run you substantially more.List price for an ECC DDR4 128GB is like close to $10K US each Nobody pays list, but still.. I’m guessing they have something like 48x256gb dimms — that is serious coin.

etaioinshrdlu 8y ago

Can one even buy an 8 socket motherboard off the shelf?

Rychard 8y ago
Once you go beyond dual socket boards (and even then, they're not widely available), you're firmly in the territory of custom form factors, and extensibility usually happens via proprietary daughterboards. At this point it's not really consumer-grade hardware though, so I wouldn't really classify it under "off the shelf".
- mmt 8y ago
  The "custom" and "proprietary" are largely irrelevant, in that they're merely internal implementation details. These systems (from SuperMicro or Lenovo, for example) still fit in a standard 19" rack, and still take standard CPUs, standard DIMMs, and standard PCIe cards. The main limitation is that the complete "barebones" system is indivisible, unlike with most 1S-2S systems, where motherboards can be mixed and matched among a wider variety of chasses. This limitation also applies to high-density offerings like blades and SM's "Twin" line. I have yet to encounter a situation where such a limitation has ever even entered the conversation.
  - Rychard 8y ago
    > The "custom" and "proprietary" are largely irrelevant, in that they're merely internal implementation details. Agreed. I was merely trying to convey that they wouldn't be the normal [E]ATX form factor boards. The parent was asking for something "off the shelf", and I took this to mean purchasing a motherboard by itself. You could get something like that from SM[0], but I doubt you'd be able to find a compatible chassis for the motherboard from anyone other than SM. At that point it probably makes more sense to just buy the system[1]. [0] http://www.supermicro.com/products/motherboard/Xeon/C600/X10QRH_.cfm http://www.supermicro.com/products/motherboard/Xeon/C600/X10... [1] http://www.supermicro.com/products/system/2U/2048/SYS-2048U-RTR4.cfm http://www.supermicro.com/products/system/2U/2048/SYS-2048U-...

mmt 8y ago

> cost of $267,512.88 per year (as calculated from another comment in the thread), I assume their profit margins for the first year are nearly non-existent NB that this was calculated based on the "All Up Front" pricing, so, for that situation, it's safer to say that their profit for the first year is $300k+ [1] and merely non-existent or negative for the second and third years. [1] Assuming $400k purchase cost as per https://news.ycombinator.com/item?id=18090058 https://news.ycombinator.com/item?id=18090058