3 ms·
Modern supercomputers essentially are large commodity clusters. Nobody builds a single fast machine ala Cray-1 anymore. It's just not scaleable past a certain
by paulmd 11y ago
Modern supercomputers essentially are large commodity clusters. Nobody builds a single fast machine ala Cray-1 anymore. It's just not scaleable past a certain point and we're long since past that. However, they still like to give them fancy names as marketing, because they cost tens of millions of dollars.
The main differentiation vs a standard datacenter is low-latency high-throughput interconnects, often with a specific or reconfigurable topology. Something like InfiniBand, where you can ensure very consistent, very favorable bounds on your latency and throughput. So think of it as being a "custom installation" or "custom cluster" instead.
Nowadays there really are only a few real choices left in high-performance computing hardware: CPU vs GPU, processor architecture (eg POWER vs x86), and interconnection. Everything else is more or less irrelevant.
I have to admit though, I've always wanted to see what you'd get if you built a Cray-2 with GPU chips (eg Pascal) or Xeon Phi as processing modules. The sheer density of chips Cray managed to cram in that case is impressive.
- thrownaway2424 11y agoWhat kind of latency do they expect from these things? Less than 50 microseconds for a small RPC or small RDMA access? I'm wondering how far these things really are from commercial cloud clusters.
- semi-extrinsic 11y agoFor a relatively modern cluster the standard is FDR Infiniband, which gives you 56 Gbit/s throughput. The latency for a small (< 32 kbytes) packet is less than 10 microseconds, down towards 1 microsecond for a 8 byte packet. Glenn Lockwood has a nice blogpost with much more detail: http://glennklockwood.blogspot.com/2013/05/fdr-infiniband-vs-dual-rail-qdr.html http://glennklockwood.blogspot.com/2013/05/fdr-infiniband-vs...
- bashinator 11y agoLess than one microsecond. https://en.wikipedia.org/wiki/InfiniBand#Performance https://en.wikipedia.org/wiki/InfiniBand#Performance
- mturmon 11y agoFor more on Cray's Aries interconnect used on the Cray XC-30 referred to in the OP, see this paper from NERSC, which has one: https://www.nersc.gov/assets/pubs_presos/NUG2014Aries.pdf https://www.nersc.gov/assets/pubs_presos/NUG2014Aries.pdf Depending on topology, you can get ~1 µs latency with Aries. It's based on PCIe-3, so it's different from InfiniBand.
- superpc 11y agoDoes this count, just curious. It is a large single machine not cluster. https://www.sgi.com/pdfs/4555.pdf https://www.sgi.com/pdfs/4555.pdf