The problem is not compute power the problem is weight and data synchronization. Each iteration or epoch builds on the previous, you either need to run the full

The problem is not compute power the problem is weight and data synchronization. Each iteration or epoch builds on the previous, you either need to run the full model on each node with part of the data and you synchronize every epoch or you run part of the model but then you need to synchronize weights after each iteration. In proof of work mining you don't need to synchronize between each iteration, that's why in mining rigs the GPUs are connected to the CPU with only a few pcie lanes (2x instead of 16x or something) and they are unsuitable for machine learning. The clusters that big orgs are using to parallelize training use extremely expensive infiniband interconnections with tens of gigabytes per second and low latency (400 Gbps in the latest oai cluster) for that reason. Unfortunately not something you can democratize anytime soon.