7 ms·

I am not sure if I should be impressed until they reveal more details. 160 TB over 40 nodes is 4 TB per node, which assuming 512 GB dimms requires only 8 dimms

I am not sure if I should be impressed until they reveal more details. 160 TB over 40 nodes is 4 TB per node, which assuming 512 GB dimms requires only 8 dimms (4 per socket) or with 256 GB dimms 16 dimms (8 per socket). How fast is the interconnect? 100Gb/s? So far it sounds like an common high memory HPC cluster unit with really unknown technical parameters.

phonon 9y ago
Where can you get 512 GB dimms?
- eveningcoffee 9y ago
  That is the assuming part because article tries to make it extremely impressive. Well, according to the link provided by pjmlp they most likely use 256 GB dimms https://youtu.be/S--Kgseuy0Q?t=450 https://youtu.be/S--Kgseuy0Q?t=450
- yuhong 9y ago
  I think even 128GB RDIMMs using TSV and 8Gbit DDR4 is expensive at this point. Interestingly, I have not even seen any 16Gbit DDR4 announcements.
pjmlp 9y ago
There was a presentation about it done last year by Keith Packard at Linux.conf.au. Hardware and Software Architecture of The Machine https://www.youtube.com/watch?v=S--Kgseuy0Q https://www.youtube.com/watch?v=S--Kgseuy0Q
Recurecur 9y ago
"So far it sounds like an common high memory HPC cluster unit with really unknown technical parameters." And really anemic CPUs to boot...
- eveningcoffee 9y ago
  Only real interesting part would be the technical details of their NGMI interconnect. Only thing that I understood so far is that it is based on FPGA that suggest that it is experimental i.e. no ASIC yet.
rbanffy 9y ago
Well... Consider https://www.supermicro.nl/products/system/7U/7088/SYS-7088B-TR4FT.cfm https://www.supermicro.nl/products/system/7U/7088/SYS-7088B-... This 7U box can have 24TB on a single 192-core/384-thread node using 128GB DIMMs. At 7U, you can fit 5 of them on 40u of rack space (assuming you can deliver power and get rid of the heat) and deal with far less inter-node communication and CPUs that can run rings around the best ARM-based ones. You can also pack 8 GPUs or your favorite Xeon Phi card (but each will behave like a cluster node) and 8 NVMe cards and 32 2.5" devices. This is slightly less dense than the HP offering, but you can order 5 of them right now. If you do, tell them I told you to. If you move it to 256GB DIMMs which, it was pointed out, is probably what HPE is using, you have 240TB per rack.
- eveningcoffee 9y ago
  The interesting part seems to be their NGMI (next -generation memory interface) that as far I understand acts as a kind of memory bus and each processor will use it more like a disk device (with special functionality like support for atomic operations etc). Unfortunately I did not find any performance information about it.
closeparen 9y ago
Isn't the idea that a single process can address all 160TB of memory as if it were local? Of course you can cobble together that much memory when you're programming a distributed system of communicating processes. The interesting part would be programming it as if for one computer.
- eveningcoffee 9y ago
  As far I understand, this memory would be not local to any of the processors - all of them access it through shared bus (fabric as they call it) and it appears to be handled rather as a weird (their words) disk device than as a memory. I did not find information about the bandwidth or latency of such architecture.
- rwmj 9y ago
  Distributed shared memory on commodity hardware/networks is a long solved problem. I was using TreadMarks back in 1994. Making it perform well, on the other hand, is a real challenge.

phonon 9y ago

Where can you get 512 GB dimms?

eveningcoffee 9y ago
That is the assuming part because article tries to make it extremely impressive. Well, according to the link provided by pjmlp they most likely use 256 GB dimms https://youtu.be/S--Kgseuy0Q?t=450 https://youtu.be/S--Kgseuy0Q?t=450
yuhong 9y ago
I think even 128GB RDIMMs using TSV and 8Gbit DDR4 is expensive at this point. Interestingly, I have not even seen any 16Gbit DDR4 announcements.

pjmlp 9y ago

There was a presentation about it done last year by Keith Packard at Linux.conf.au. Hardware and Software Architecture of The Machine https://www.youtube.com/watch?v=S--Kgseuy0Q https://www.youtube.com/watch?v=S--Kgseuy0Q

Recurecur 9y ago

"So far it sounds like an common high memory HPC cluster unit with really unknown technical parameters." And really anemic CPUs to boot...

eveningcoffee 9y ago
Only real interesting part would be the technical details of their NGMI interconnect. Only thing that I understood so far is that it is based on FPGA that suggest that it is experimental i.e. no ASIC yet.

rbanffy 9y ago

Well... Consider https://www.supermicro.nl/products/system/7U/7088/SYS-7088B-TR4FT.cfm https://www.supermicro.nl/products/system/7U/7088/SYS-7088B-... This 7U box can have 24TB on a single 192-core/384-thread node using 128GB DIMMs. At 7U, you can fit 5 of them on 40u of rack space (assuming you can deliver power and get rid of the heat) and deal with far less inter-node communication and CPUs that can run rings around the best ARM-based ones. You can also pack 8 GPUs or your favorite Xeon Phi card (but each will behave like a cluster node) and 8 NVMe cards and 32 2.5" devices. This is slightly less dense than the HP offering, but you can order 5 of them right now. If you do, tell them I told you to. If you move it to 256GB DIMMs which, it was pointed out, is probably what HPE is using, you have 240TB per rack.

eveningcoffee 9y ago
The interesting part seems to be their NGMI (next -generation memory interface) that as far I understand acts as a kind of memory bus and each processor will use it more like a disk device (with special functionality like support for atomic operations etc). Unfortunately I did not find any performance information about it.

closeparen 9y ago

Isn't the idea that a single process can address all 160TB of memory as if it were local? Of course you can cobble together that much memory when you're programming a distributed system of communicating processes. The interesting part would be programming it as if for one computer.

eveningcoffee 9y ago
As far I understand, this memory would be not local to any of the processors - all of them access it through shared bus (fabric as they call it) and it appears to be handled rather as a weird (their words) disk device than as a memory. I did not find information about the bandwidth or latency of such architecture.
rwmj 9y ago
Distributed shared memory on commodity hardware/networks is a long solved problem. I was using TreadMarks back in 1994. Making it perform well, on the other hand, is a real challenge.