4 ms·

While currently it seems like the only (?) InfiniBand vendor is Nvida/Mellanox, there used to be more folks selling the gear. For example Intel used to sell swi

While currently it seems like the only (?) InfiniBand vendor is Nvida/Mellanox, there used to be more folks selling the gear. For example Intel used to sell switches (had some for an Isilon backend): * https://www.intel.com/content/dam/support/us/en/documents/network-and-i-o/fabric-products/TrueScale12000Switch_HWInstallGuide_G91928_04.pdf https://www.intel.com/content/dam/support/us/en/documents/ne... (PDF) Officially there are a bunch of folks in the IB alliance: * https://www.infinibandta.org/member-listing/ https://www.infinibandta.org/member-listing/

jauntywundrkind 1y ago
Intel's Omni-Path was very similar but a little different from Infiniband. Anyone remember any details on how; I'm forgetting? Omni-Path was sold off to Cornelis Networks in 2020. Cornelis just released new 400Gbps products last week, their CN5000 line, with 48 and 576 port switches. https://www.crn.com/news/components-peripherals/intel-spins-out-omni-path-interconnect-business-into-stand-alone-company https://www.crn.com/news/components-peripherals/intel-spins-...
- throw0101d 1y ago
  > Intel's Omni-Path was very similar but a little different from Infiniband. While Intel may have had the 'IB-adjacent' Omni-Path, they also had plain-old IB as well. I ran these/similar switches for an Isilon back-end network (starting when Isilon was still an independent company, pre-EMC buyout, pre-Dell buyout, in the OneFS 6.x days): * https://www.dell.com/support/kbdoc/en-us/000019215/install-and-hardware-upgrade-tips-for-intel-12800-ib-switch https://www.dell.com/support/kbdoc/en-us/000019215/install-a...
  - rincebrain 1y ago
    Intel bought Qlogic's IB division when they sold it off, in 2012; I believe at one point the former QLogic parts were branded Omni-Path before it diverged. (You can still see the DNA of that in how the Omni-Path driver stack is a relative of the Qlogic drivers, last I knew.) Before that, Mellanox ate Voltaire, who was the other large vendor in the IB space. So at this point, I believe NVIDIA's Mellanox devices are the only people selling IB chips these days, and glancing at the TOP500 from June, seems like a good half (45.8%) of the supercomputers listed there are using either Ethernet-based or otherwise non-IB stacks.
    - throw0101d 1y ago
      > […] seems like a good half (45.8%) of the supercomputers listed there are using either Ethernet-based or otherwise non-IB stacks. I've done HPC for the last little while, but mostly Eth-based. Lately I have done a smattering more of IB, and (AFIACT) the main advantage IB gets you is lower latency (both 'innately' at the network layer, and with more 'out-of-box' RDMA).
      - rincebrain 1y ago
        One reason we preferred Ethernet, when I last cared an age ago, was that we had a longstanding investment in building a lot of Windows-based bespoke software, and Infiniband support on Windows might now be up to merely 'iffy' instead of "requires a cold boot after you try using RDMA". So after we tried using IB once and got burned by being promised the software would be there Soon(tm), we went with then-10GbE for the portion we managed. Infiniband is, if not purpose-built for HPC, at least more specifically suited for it. Ethernet mostly gets to be usable for HPC setups, IIRC, by implementing IB verbs over Ethernet and some Ethernet extensions that let it pretend _really_ hard that it's a guaranteed delivery transport. But yes, I think IB's main advantage these days is reliably lower latency (which some kinds of HPC workloads certainly care more about). I believe you also used to get better performance on large switch fabrics when you started saturating significant fractions of the links, but I don't know that that's true any more, since some markets of Ethernet switch started caring about that and listing it as a bullet point.
deleted 1y ago
[deleted]

jauntywundrkind 1y ago

Intel's Omni-Path was very similar but a little different from Infiniband. Anyone remember any details on how; I'm forgetting? Omni-Path was sold off to Cornelis Networks in 2020. Cornelis just released new 400Gbps products last week, their CN5000 line, with 48 and 576 port switches. https://www.crn.com/news/components-peripherals/intel-spins-out-omni-path-interconnect-business-into-stand-alone-company https://www.crn.com/news/components-peripherals/intel-spins-...

throw0101d 1y ago
> Intel's Omni-Path was very similar but a little different from Infiniband. While Intel may have had the 'IB-adjacent' Omni-Path, they also had plain-old IB as well. I ran these/similar switches for an Isilon back-end network (starting when Isilon was still an independent company, pre-EMC buyout, pre-Dell buyout, in the OneFS 6.x days): * https://www.dell.com/support/kbdoc/en-us/000019215/install-and-hardware-upgrade-tips-for-intel-12800-ib-switch https://www.dell.com/support/kbdoc/en-us/000019215/install-a...
- rincebrain 1y ago
  Intel bought Qlogic's IB division when they sold it off, in 2012; I believe at one point the former QLogic parts were branded Omni-Path before it diverged. (You can still see the DNA of that in how the Omni-Path driver stack is a relative of the Qlogic drivers, last I knew.) Before that, Mellanox ate Voltaire, who was the other large vendor in the IB space. So at this point, I believe NVIDIA's Mellanox devices are the only people selling IB chips these days, and glancing at the TOP500 from June, seems like a good half (45.8%) of the supercomputers listed there are using either Ethernet-based or otherwise non-IB stacks.
  - throw0101d 1y ago
    > […] seems like a good half (45.8%) of the supercomputers listed there are using either Ethernet-based or otherwise non-IB stacks. I've done HPC for the last little while, but mostly Eth-based. Lately I have done a smattering more of IB, and (AFIACT) the main advantage IB gets you is lower latency (both 'innately' at the network layer, and with more 'out-of-box' RDMA).
    - rincebrain 1y ago
      One reason we preferred Ethernet, when I last cared an age ago, was that we had a longstanding investment in building a lot of Windows-based bespoke software, and Infiniband support on Windows might now be up to merely 'iffy' instead of "requires a cold boot after you try using RDMA". So after we tried using IB once and got burned by being promised the software would be there Soon(tm), we went with then-10GbE for the portion we managed. Infiniband is, if not purpose-built for HPC, at least more specifically suited for it. Ethernet mostly gets to be usable for HPC setups, IIRC, by implementing IB verbs over Ethernet and some Ethernet extensions that let it pretend _really_ hard that it's a guaranteed delivery transport. But yes, I think IB's main advantage these days is reliably lower latency (which some kinds of HPC workloads certainly care more about). I believe you also used to get better performance on large switch fabrics when you started saturating significant fractions of the links, but I don't know that that's true any more, since some markets of Ethernet switch started caring about that and listing it as a bullet point.

deleted 1y ago

[deleted]