Yes. You'd have to package up slices of the training set and the prior epoch as work units and host them. Our hypothetical GPT@home clients download the trainin

Yes. You'd have to package up slices of the training set and the prior epoch as work units and host them. Our hypothetical GPT@home clients download the training set, train just those examples, and then send back their gradients. The server then averages all the gradients together. As far as I can tell this isn't any different conceptually from the distributed inference setup that PyTorch/Huggingface Accelerate/etc support. Of course those were built for cooperating machines owned by the same organization with lots of high-speed I/O. In an @home/BOINC setup, you need to package things up into work units that get sent out to other people's computers, and you don't get anything back until the end. So it's higher-latency and you have to worry about people cheating the system with bogus work. I'm not sure if this would be more efficient or cheaper than just renting out a bunch of AWS spot instances with GPUs in them.