If you want to split tensorwise yes. Layerwise splits could go over Ethernet. I would be interested to see how feasible hybrid approaches would be, e.g. connec

If you want to split tensorwise yes. Layerwise splits could go over Ethernet. I would be interested to see how feasible hybrid approaches would be, e.g. connect each pair up directly via ConnectX and then connect the sets together via Ethernet.