4 ms·

How does IB compare to Cray’s new interconnect? All supercomputing nodes these days appear to be water cooled, high density servers with a number of Nvidia GPU

by jmpman 7y ago

How does IB compare to Cray’s new interconnect? All supercomputing nodes these days appear to be water cooled, high density servers with a number of Nvidia GPUs connected to a low latency interconnect fabric, along with a Lustre/GPFS storage cluster. The interconnect fabric used to be the secret sauce, and IB has commoditized it. Why buy Cray?

baybal2 7y ago
"Why buy Accenture?" I feel, it is the same type "aspirational" acquisition
zamadatix 7y ago
Likely it's mostly just a customer base grab. I will say there is more to an interconnect than "is the hardware commodity or not" though, you have to be able to show you are able to come up with ways to use it more efficiently than competitor designs.
aetimmes 7y ago
Cray just won a $600M bid for Oak Ridge National Labs' new exaflop HPC cluster using AMD Epyc/Radeon. Most IB cards come from Intel and Mellanox (recently purchased by NVidia). IB is a commodity, sure, but it's becoming increasingly tied into the Intel/NVidia ecosystem, and having the ability to go with another interconnect/processor architecture is a powerful strategic advantage.
- gnufx 7y ago
  As far as I know, Intel no longer does the IB-compatiable Truscale, or whatever it was called finally. It's OPA now.
- pinewurst 7y ago
  It’s funny then that Omni-Path originated with Intel’s purchase of the Cray interconnect team back in 2012. https://newsroom.intel.com/news-releases/intel-acquires-industry-leading-high-performance-computing-interconnect-technology-and-expertise/#gs.cmt9dr https://newsroom.intel.com/news-releases/intel-acquires-indu...
  - BooneJS 7y ago
    And the Q-Logic IB business.
jabl 7y ago
You're correct in that, from the HW side, the "crown jewels" of Cray is the network ASIC's. Everything else is pretty much commodity. AFAICS, the main difference between the latest Cray network (slinghshot) and IB are: - Adaptive non-minimal routing, needed to get good performance on their more scalable network topology. IB has basically static ECMP, which works good for fat trees, which scales as O(log N). But for really large machines topologies like Dragonfly which Cray is using become interesting. - QoS features to isolate jobs from each other. - Some kind of compatibility with Ethernet. Not sure exactly how this works, but apparently when the NIC communicates with other compute nodes in the cluster it can use the high-performance mode, but if it communicates with some other system, such as Lustre servers, or management systems, or maybe even, say, pulling a container image from an external system, it looks and behaves like an ethernet NIC with the normal Linux TCP/IP stack running on top.
gnufx 7y ago
I don't think that "all" is true, and I'm not sure what "commoditized" means for IB more than any other HPC interconnect. They're all single-vendor as far as I know (as I don't think you can just buy Oracle IB): IB, OPA, Aries (and Slingshot), TOFU, BXI, and maybe others -- I don't know what the Chinese ones actually are.
- mappu 7y ago
  Gen-Z interconnect is multi-vendor (including Cray and HPE). There was a talk about it at LCA: https://www.youtube.com/watch?v=MSCvkU0UKIU https://www.youtube.com/watch?v=MSCvkU0UKIU

by jmpman 7y ago

baybal2 7y ago

"Why buy Accenture?" I feel, it is the same type "aspirational" acquisition

zamadatix 7y ago

Likely it's mostly just a customer base grab. I will say there is more to an interconnect than "is the hardware commodity or not" though, you have to be able to show you are able to come up with ways to use it more efficiently than competitor designs.

aetimmes 7y ago

Cray just won a $600M bid for Oak Ridge National Labs' new exaflop HPC cluster using AMD Epyc/Radeon. Most IB cards come from Intel and Mellanox (recently purchased by NVidia). IB is a commodity, sure, but it's becoming increasingly tied into the Intel/NVidia ecosystem, and having the ability to go with another interconnect/processor architecture is a powerful strategic advantage.

gnufx 7y ago
As far as I know, Intel no longer does the IB-compatiable Truscale, or whatever it was called finally. It's OPA now.
pinewurst 7y ago
It’s funny then that Omni-Path originated with Intel’s purchase of the Cray interconnect team back in 2012. https://newsroom.intel.com/news-releases/intel-acquires-industry-leading-high-performance-computing-interconnect-technology-and-expertise/#gs.cmt9dr https://newsroom.intel.com/news-releases/intel-acquires-indu...
- BooneJS 7y ago
  And the Q-Logic IB business.

jabl 7y ago

You're correct in that, from the HW side, the "crown jewels" of Cray is the network ASIC's. Everything else is pretty much commodity. AFAICS, the main difference between the latest Cray network (slinghshot) and IB are: - Adaptive non-minimal routing, needed to get good performance on their more scalable network topology. IB has basically static ECMP, which works good for fat trees, which scales as O(log N). But for really large machines topologies like Dragonfly which Cray is using become interesting. - QoS features to isolate jobs from each other. - Some kind of compatibility with Ethernet. Not sure exactly how this works, but apparently when the NIC communicates with other compute nodes in the cluster it can use the high-performance mode, but if it communicates with some other system, such as Lustre servers, or management systems, or maybe even, say, pulling a container image from an external system, it looks and behaves like an ethernet NIC with the normal Linux TCP/IP stack running on top.

gnufx 7y ago

I don't think that "all" is true, and I'm not sure what "commoditized" means for IB more than any other HPC interconnect. They're all single-vendor as far as I know (as I don't think you can just buy Oracle IB): IB, OPA, Aries (and Slingshot), TOFU, BXI, and maybe others -- I don't know what the Chinese ones actually are.

mappu 7y ago
Gen-Z interconnect is multi-vendor (including Cray and HPE). There was a talk about it at LCA: https://www.youtube.com/watch?v=MSCvkU0UKIU https://www.youtube.com/watch?v=MSCvkU0UKIU