Whatever they say, Omni-Path isn't (compatible with) Infiniband just because it supports the Openfabrics interface, like some Ethernet technologies. Qlogic's I

Whatever they say, Omni-Path isn't (compatible with) Infiniband just because it supports the Openfabrics interface, like some Ethernet technologies. Qlogic's Infinipath was, so you could mix Mellanox HCAs and Qlogic switches, for instance; I assume that also applies to Truscale, but I haven't used it. DFT programs like VASP are likely dominated by time in mpi_alltoallv at scale, and probably depend on the implementation of that more than the fabric, assuming similar latencies. I don't know how/if that's helped by offloading; I (and a colleague at another site) couldn't make a previous generation of the support work properly and promises of help never materialized. Also, real usage may not reflect IMB-type mpi_alltoallv benchmarks. (VASP is a particularly unfortunate example, since the licence requires you to be a member of a specific research group, as opposed to general support staff or, as far as I could see when I read a copy, vendors.) At least without the complication of offloading, according to http://simgrid.gforge.inria.fr/ http://simgrid.gforge.inria.fr/ a few P2P metrics characterize the network well for simulating real MPI programs/libraries.