I’m pretty confused because your response seems very obviously confused. Firstly, the latency problem in training is indeed not the same as that of evaluation.

I’m pretty confused because your response seems very obviously confused. Firstly, the latency problem in training is indeed not the same as that of evaluation. The cycle looks like 1. Input model weights, 2. Choose some subsets of training data, 3. Update weights based on model evaluated against, 4. Back to step 1. When you do distributed training, step 4 requires collecting updated weights from each worker and then merging them together. You want some limits on the amount of computation between each merge because the merge is less useful when the parts have diverged further (imagine some super simple example where a model has two neurones in a layer and they tend to become sensitive to situations A and B. If you distribute your training and some come back with neurone one becoming strongly sensitive to case A and others with it strongly sensitive to case B, and you merge them together, you likely get a fuzzy thing where the first is not very sensitive to either case yet, potentially throwing away a lot of the information that was gained in training. To put some numbers on that, let’s say you have some 1B parameter model fast home network at 300 Mbps with 2ms latency to whatever is doing the coordination (ie probably in the same city). Maybe with some clever compression or dropout layers you cause an update to be 1GB. That’s then a little over a minute to send your weights out for coordination and get back 1GB of fresh weights (probably this would be larger than 1GB and this ignores any synchronisation with other nodes). If you compare this to the networking fabric you might expect in a GPU cluster, that’s something like 200Gbps and a few us of latency so even sending 4GB of weights back and forth (makes more sense as compression costs compute time) is more like 200ms plus latency. Note that the difference when trying to be favourable to your home setup is >300x. Secondly, let’s look at compute. A ‘smallish’ cluster of GPUs might look like 8 H100s in some very specialised box, plus a bunch of extra supporting hardware. That’ll give you over 7e15 half-precision flops. In your train@home example, someone with a top-of-the-line prosumer GPU might get something like 1e14 half-precision flops, though I think you’d probably be pretty lucky to get 1/5 of that. So it takes 70 (more like 350) of these machines to match the flops of the smallish cluster, and that ignores differences in memory bandwidth and in the coordination of moving more data around (because each node is smaller) over slower links. To me, it seems the main things that make it unviable are the coordination and the network requirements, but maybe something sufficiently clever could someday make this kind of distribution more viable. If you compare this whole thing to running Monte Carlo algorithms like folding@home, hopefully you can see the difference in that the size of the problem description is a lot smaller and synchronisation isn’t really required at all. And didn’t folding@home largely fail anyway?