One reason we preferred Ethernet, when I last cared an age ago, was that we had a longstanding investment in building a lot of Windows-based bespoke software, a

One reason we preferred Ethernet, when I last cared an age ago, was that we had a longstanding investment in building a lot of Windows-based bespoke software, and Infiniband support on Windows might now be up to merely 'iffy' instead of "requires a cold boot after you try using RDMA". So after we tried using IB once and got burned by being promised the software would be there Soon(tm), we went with then-10GbE for the portion we managed. Infiniband is, if not purpose-built for HPC, at least more specifically suited for it. Ethernet mostly gets to be usable for HPC setups, IIRC, by implementing IB verbs over Ethernet and some Ethernet extensions that let it pretend _really_ hard that it's a guaranteed delivery transport. But yes, I think IB's main advantage these days is reliably lower latency (which some kinds of HPC workloads certainly care more about). I believe you also used to get better performance on large switch fabrics when you started saturating significant fractions of the links, but I don't know that that's true any more, since some markets of Ethernet switch started caring about that and listing it as a bullet point.