I'm not an expert in distributed computing so take what I say with a grain of salt, but I believe it would be really hard. The premise of all the @home project

I'm not an expert in distributed computing so take what I say with a grain of salt, but I believe it would be really hard. The premise of all the @home projects is that each compute unit can do some relatively large amount of work in isolation and then communicate the result back to a central server. If one of the client disconnects in the middle of it's computation, it can just be redistributed to another client. And it doesn't really matter if a client is very slow. For the same reason network latency barely matters in those scenarios. On the other hand, to train GPT-3 the model is split into many GPUs at every level, even a simple matrix multiplication goes accross the network, and there is no way to continue the computation until every GPU has communicated their result. So the issue here is that if a compute unit is very slow, then all the other ones* will block waiting for it, so your total cluster speed is defined by the slowest unit + slowest network time. And if a unit of compute disconnects (which happens constantly in @home scenarios), then its work need to be attributed to another unit and re-executed, while every other unit is idle waiting for that to happen. I'm sure there are smarter ways to split such a model than just trying to replicate what we currently do on big compute clusters, but it's definitly a hard problem. Finding better ways to do distributed training with less communication between units is an active area of research *: Not exactly all the other ones, because you could divide your cluster into chunks, each chunk handling one batch of data. So the chunks would be independents to a certain degree. But the size of the batch has an impact on the final performance of the model (because of batch normalization operation) so you still want it to be quite big, which requires many clients for a single chunk.