For more on Cray's Aries interconnect used on the Cray XC-30 referred to in the OP, see this paper from NERSC, which has one: https://www.nersc.gov/assets/pubs_

For more on Cray's Aries interconnect used on the Cray XC-30 referred to in the OP, see this paper from NERSC, which has one: https://www.nersc.gov/assets/pubs_presos/NUG2014Aries.pdf https://www.nersc.gov/assets/pubs_presos/NUG2014Aries.pdf Depending on topology, you can get ~1 µs latency with Aries. It's based on PCIe-3, so it's different from InfiniBand.