However much Ethernet has changed, it's not appropriate for typical HPC use (c.f. IB, OPA, Aries, ...). I investigated Cisco's UCS at some length, for instance

However much Ethernet has changed, it's not appropriate for typical HPC use (c.f. IB, OPA, Aries, ...). I investigated Cisco's UCS at some length, for instance, which they had persuaded an important person to buy, and that's at least partly the baby of an MPI person. (I recall Infiniband was designed as a "datacentre" thing.)