4 ms·

Intel bought Qlogic's IB division when they sold it off, in 2012; I believe at one point the former QLogic parts were branded Omni-Path before it diverged. (You

Intel bought Qlogic's IB division when they sold it off, in 2012; I believe at one point the former QLogic parts were branded Omni-Path before it diverged. (You can still see the DNA of that in how the Omni-Path driver stack is a relative of the Qlogic drivers, last I knew.) Before that, Mellanox ate Voltaire, who was the other large vendor in the IB space. So at this point, I believe NVIDIA's Mellanox devices are the only people selling IB chips these days, and glancing at the TOP500 from June, seems like a good half (45.8%) of the supercomputers listed there are using either Ethernet-based or otherwise non-IB stacks.

throw0101d 1y ago
> […] seems like a good half (45.8%) of the supercomputers listed there are using either Ethernet-based or otherwise non-IB stacks. I've done HPC for the last little while, but mostly Eth-based. Lately I have done a smattering more of IB, and (AFIACT) the main advantage IB gets you is lower latency (both 'innately' at the network layer, and with more 'out-of-box' RDMA).
- rincebrain 1y ago
  One reason we preferred Ethernet, when I last cared an age ago, was that we had a longstanding investment in building a lot of Windows-based bespoke software, and Infiniband support on Windows might now be up to merely 'iffy' instead of "requires a cold boot after you try using RDMA". So after we tried using IB once and got burned by being promised the software would be there Soon(tm), we went with then-10GbE for the portion we managed. Infiniband is, if not purpose-built for HPC, at least more specifically suited for it. Ethernet mostly gets to be usable for HPC setups, IIRC, by implementing IB verbs over Ethernet and some Ethernet extensions that let it pretend _really_ hard that it's a guaranteed delivery transport. But yes, I think IB's main advantage these days is reliably lower latency (which some kinds of HPC workloads certainly care more about). I believe you also used to get better performance on large switch fabrics when you started saturating significant fractions of the links, but I don't know that that's true any more, since some markets of Ethernet switch started caring about that and listing it as a bullet point.

throw0101d 1y ago

> […] seems like a good half (45.8%) of the supercomputers listed there are using either Ethernet-based or otherwise non-IB stacks. I've done HPC for the last little while, but mostly Eth-based. Lately I have done a smattering more of IB, and (AFIACT) the main advantage IB gets you is lower latency (both 'innately' at the network layer, and with more 'out-of-box' RDMA).

rincebrain 1y ago
One reason we preferred Ethernet, when I last cared an age ago, was that we had a longstanding investment in building a lot of Windows-based bespoke software, and Infiniband support on Windows might now be up to merely 'iffy' instead of "requires a cold boot after you try using RDMA". So after we tried using IB once and got burned by being promised the software would be there Soon(tm), we went with then-10GbE for the portion we managed. Infiniband is, if not purpose-built for HPC, at least more specifically suited for it. Ethernet mostly gets to be usable for HPC setups, IIRC, by implementing IB verbs over Ethernet and some Ethernet extensions that let it pretend _really_ hard that it's a guaranteed delivery transport. But yes, I think IB's main advantage these days is reliably lower latency (which some kinds of HPC workloads certainly care more about). I believe you also used to get better performance on large switch fabrics when you started saturating significant fractions of the links, but I don't know that that's true any more, since some markets of Ethernet switch started caring about that and listing it as a bullet point.