4 ms·
GPT-J and GPT-Neo are previous open source reproductions of GPT, but distributed training is overrated. There’s way too much communications overhead. I’m not su
by astrange 5y ago
GPT-J and GPT-Neo are previous open source reproductions of GPT, but distributed training is overrated. There’s way too much communications overhead. I’m not sure any of the various @home projects have ever accomplished much for that matter.
Instead they use donated time on a TPU or a GPU cloud provider.
An exception is Leela Zero since you can realistically have it play chess at home.
- airstrike 5y ago> Instead they use donated time on a TPU or a GPU cloud provider. So instead distribute the funding? And distributing running models too? Something something crowdfunding blockchain cryptocurrency Or alternatively https://www.microsoft.com/en-us/research/publication/decentralized-collaborative-ai-on-blockchain/ https://www.microsoft.com/en-us/research/publication/decentr...
- kache_ 5y ago>donated time on a TPU or a GPU cloud provider hmmm... what about people who mine crypto on their hardware, and then that hardware goes into a pool of shared $ used to rent out a TPU