Depends on your bisection bandwidth requirement and software cost. If you want to do MPI it is going to be cheaper to buy some InfiniBand cards. But if you have

Depends on your bisection bandwidth requirement and software cost. If you want to do MPI it is going to be cheaper to buy some InfiniBand cards. But if you have a small cluster and tightly coupled code, or large I/O requirements, then a PCIe switch might be the go. These get used on some video on demand systems and signal processing architectures: https://www.microsemi.com/products/drivers-interfaces-and-pcie-switches/pcie-switches/pcie-fanout-switches/pm8536-pfx-96xg3 https://www.microsemi.com/products/drivers-interfaces-and-pc... The Dolphin systems are tuned better for computation: http://www.dolphinics.com/products/IXS600.html http://www.dolphinics.com/products/IXS600.html I expect small PCIe NVMe external storage systems to become quite common in the near future, because enterprise systems need multipath storage for reliability; 8GB/s FC is too slow for SSDs, let alone NVMe, same with SAS bus expanders. https://events.linuxfoundation.org/sites/events/files/slides/LinuxVault2015_KeithBusch_PCIeMPath.pdf https://events.linuxfoundation.org/sites/events/files/slides...