5 ms·
What kind of latency do they expect from these things? Less than 50 microseconds for a small RPC or small RDMA access? I'm wondering how far these things real
by thrownaway2424 11y ago
What kind of latency do they expect from these things? Less than 50 microseconds for a small RPC or small RDMA access? I'm wondering how far these things really are from commercial cloud clusters.
- semi-extrinsic 11y agoFor a relatively modern cluster the standard is FDR Infiniband, which gives you 56 Gbit/s throughput. The latency for a small (< 32 kbytes) packet is less than 10 microseconds, down towards 1 microsecond for a 8 byte packet. Glenn Lockwood has a nice blogpost with much more detail: http://glennklockwood.blogspot.com/2013/05/fdr-infiniband-vs-dual-rail-qdr.html http://glennklockwood.blogspot.com/2013/05/fdr-infiniband-vs...
- bashinator 11y agoLess than one microsecond. https://en.wikipedia.org/wiki/InfiniBand#Performance https://en.wikipedia.org/wiki/InfiniBand#Performance
- mturmon 11y agoFor more on Cray's Aries interconnect used on the Cray XC-30 referred to in the OP, see this paper from NERSC, which has one: https://www.nersc.gov/assets/pubs_presos/NUG2014Aries.pdf https://www.nersc.gov/assets/pubs_presos/NUG2014Aries.pdf Depending on topology, you can get ~1 µs latency with Aries. It's based on PCIe-3, so it's different from InfiniBand.