I wasn't talking about MPI, but if you want to talk about MPI, note that OmniPath gets faster as you have more cores talking to it, and it takes full advantage

I wasn't talking about MPI, but if you want to talk about MPI, note that OmniPath gets faster as you have more cores talking to it, and it takes full advantage of delivering messages to the memory near the receiving process, so you'd better not move threads or processes very far. If you look in your HPC historybook, you'll notice that I was the system architect for InfiniPath.