4 ms·

PCIe switching sounds nice, but let me tell you that it is a terrible nightmare on every level. Hardware: these PCIe switches are finicky as all get out; will

PCIe switching sounds nice, but let me tell you that it is a terrible nightmare on every level. Hardware: these PCIe switches are finicky as all get out; will it link up to your device reliably? Who knows, depends on how the switch and device manufacturers interpreted the PCIe specs. Software: there is no nice easy standard for connecting host-to-host (which I’m assuming is what one would want on a multinode system Like the one linked). There exists the idea of a PCIe non-transparent bridge, which if I recall correctly, allows CPUs on either side of it to do blind memory reads and writes to each other (hope your IOMMU is set up correctly to protect you from ugliness that could occur here). Long story short there’s a reason PCIe is a local bus; it was never designed to do anything else. Anything else is largely a proprietary bolt on.

nsky-world 6y ago
Totally agree
rektide 6y ago
I was definitely thinking of NTB, for host-to-host connectivity. As you say, it doesn't provide a whole lot of capabilities. However, there's been a pretty competent ethernet driver available on Linux since Linux 3.9 released April 2013. There's out of kernel NTRDMA if you want ibverbs like communication that will make really good use of the PCIe bus too. In addition to mapped memory, there are some other modest capabilities: support for ringing the peer's doorbell (one way), & scratchpad configuration. There's a pretty good 2016 presentation on NTB in Linux here: https://events.static.linuxfound.org/sites/events/files/slides/Linux%20NTB_0.pdf https://events.static.linuxfound.org/sites/events/files/slid... Interesting to note, Nvidia has docs for settings up NTB on their Pegasus & Xavier systems: https://docs.nvidia.com/drive/drive_os_5.1.6.1L/nvvib_docs/index.html#page/DRIVE_OS_Linux_SDK_Development_Guide/System%20Programming/sys_components_non_transparent_bridging.html https://docs.nvidia.com/drive/drive_os_5.1.6.1L/nvvib_docs/i... I for one think more people should be investigating & harnessing the open-source Linux NTB tools. Switches are a bit expensive, but compared to ethernet switches their cost/throughput is great. Alas, yes, the whole ecosystem is woefully undersupported, under-developed. Isn't PCI part of the whole PC thing, this great revolution in computing that allowed interoperability to rise? Such a pity that all the PCI SIG's seemed to fizzle out, that not a lot of people participated. Sure we got some SR-IOV but that feels like as far as we got.

nsky-world 6y ago

Totally agree

rektide 6y ago

I was definitely thinking of NTB, for host-to-host connectivity. As you say, it doesn't provide a whole lot of capabilities. However, there's been a pretty competent ethernet driver available on Linux since Linux 3.9 released April 2013. There's out of kernel NTRDMA if you want ibverbs like communication that will make really good use of the PCIe bus too. In addition to mapped memory, there are some other modest capabilities: support for ringing the peer's doorbell (one way), & scratchpad configuration. There's a pretty good 2016 presentation on NTB in Linux here: https://events.static.linuxfound.org/sites/events/files/slides/Linux%20NTB_0.pdf https://events.static.linuxfound.org/sites/events/files/slid... Interesting to note, Nvidia has docs for settings up NTB on their Pegasus & Xavier systems: https://docs.nvidia.com/drive/drive_os_5.1.6.1L/nvvib_docs/index.html#page/DRIVE_OS_Linux_SDK_Development_Guide/System%20Programming/sys_components_non_transparent_bridging.html https://docs.nvidia.com/drive/drive_os_5.1.6.1L/nvvib_docs/i... I for one think more people should be investigating & harnessing the open-source Linux NTB tools. Switches are a bit expensive, but compared to ethernet switches their cost/throughput is great. Alas, yes, the whole ecosystem is woefully undersupported, under-developed. Isn't PCI part of the whole PC thing, this great revolution in computing that allowed interoperability to rise? Such a pity that all the PCI SIG's seemed to fizzle out, that not a lot of people participated. Sure we got some SR-IOV but that feels like as far as we got.