4 ms·
Transformers are parallelize-able, right? What’s stopping a large group of people from pooling their compute power together and working towards something like t
by Mockapapella 4y ago
Transformers are parallelize-able, right? What’s stopping a large group of people from pooling their compute power together and working towards something like this? IIRC there were some crypto projects a while back that we’re trying to create something similar (golem?)
- visarga 4y agoThere are the Eleuther.ai and BigScience projects working on public foundation models. They have a few releases already and currently training GPT-3 sized models.
- joshcryer 4y agoThere are people working on reproducing the models, see here for Dall-E 2 for example: https://github.com/lucidrains/DALLE2-pytorch https://github.com/lucidrains/DALLE2-pytorch It's often not worth it to decentralize the computation of the trained model though but it's not hard to get donated cycles and groups are working on it. Don't fret because Google isn't releasing the API/code. They released the paper and that's all you need.
- sinenomine 4y agoYou really need a decent infiniband-linked cluster to train large models.