The cost of moving data from one gpu to the next will destroy performance. The system are moving in the opposite direction (look at Dojo architecture or TensTo

The cost of moving data from one gpu to the next will destroy performance. The system are moving in the opposite direction (look at Dojo architecture or TensTorrent) The silver lining is that the cost of training will fall substantially with those architecture that are not based in reusing gpu.