4 ms·
> So what we really need, are ways to train models in a distributed fashion. How is the replication effort for this project handling that? They have over 600 p
by leereeves 5y ago
> So what we really need, are ways to train models in a distributed fashion.
How is the replication effort for this project handling that? They have over 600 people on their Discord right now, do they have some way for them to cooperate on training a single model?
- sva_ 5y agoAs far as I can see, No.
- astrange 5y agoGPT-J and GPT-Neo are previous open source reproductions of GPT, but distributed training is overrated. There’s way too much communications overhead. I’m not sure any of the various @home projects have ever accomplished much for that matter. Instead they use donated time on a TPU or a GPU cloud provider. An exception is Leela Zero since you can realistically have it play chess at home.
- airstrike 5y ago> Instead they use donated time on a TPU or a GPU cloud provider. So instead distribute the funding? And distributing running models too? Something something crowdfunding blockchain cryptocurrency Or alternatively https://www.microsoft.com/en-us/research/publication/decentralized-collaborative-ai-on-blockchain/ https://www.microsoft.com/en-us/research/publication/decentr...
- kache_ 5y ago>donated time on a TPU or a GPU cloud provider hmmm... what about people who mine crypto on their hardware, and then that hardware goes into a pool of shared $ used to rent out a TPU