3 ms·
Is there ever going to be a way to distribute training - I would think this is the only way open models will eventually be able to exist and not just be owned b
by andy_ppp 4y ago
Is there ever going to be a way to distribute training - I would think this is the only way open models will eventually be able to exist and not just be owned by Microsoft, Google, Facebook and AWS.
- mrdoops 4y agoThere almost surely will and Elixir/OTP is going to do it best.
- nerdponx 4y agoTraining is distributed already, but over a big cluster of machine in a data center. I've always wanted there to be something like BOINC/Gridcoin for fitting these giant neural networks.
- jessfyi 4y agoThe BigScience team (a working group of researchers that trained the BLOOM-176B LLM last year) released Petals [0][1] which allows distributed inference and fine-tuning of BLOOM, with the option to pick a custom model + private swarm. SWARM [2][3] is a WIP from yandex and UW that shares some of the same codebase, but is for distributed training. [0] https://petals.ml/ https://petals.ml/ [1] https://github.com/bigscience-workshop/petals https://github.com/bigscience-workshop/petals [2] https://github.com/yandex-research/swarm https://github.com/yandex-research/swarm [3] https://twitter.com/m_ryabinin/status/1625175933492641814 https://twitter.com/m_ryabinin/status/1625175933492641814