3 ms·

Out of curiosity, why aren't we crowd sourcing distributed training of LLMs where anyone can join by bringing their hardware or data? Moreover find a way to inc

Out of curiosity, why aren't we crowd sourcing distributed training of LLMs where anyone can join by bringing their hardware or data? Moreover find a way to incorporate this into a blockchain so there is full transparency but also add in differential privacy to protect every participant. Am I being too crazy here?

albertzeyer 4y ago
There is the Open Assistant project: https://github.com/LAION-AI/Open-Assistant https://github.com/LAION-AI/Open-Assistant There is also EleutherAI (https://www.eleuther.ai/about/ https://www.eleuther.ai/about/) with GPT-NeoX (https://github.com/EleutherAI/gpt-neox https://github.com/EleutherAI/gpt-neox).
moffkalast 4y ago
Just make sure it's written in Rust, uses a Sveltekit frontend and <some other buzzwords I can't remember right now>.
- wg0 4y ago
  And SQLite as local cache with CRDTs enabled whereas everything else from text search to queuing on PostgreSQL?
nodja 4y ago
https://petals.ml/ https://petals.ml/
- Miraste 4y ago
  Petals doesn't train new models, it only runs BLOOM in a distributed way.
  - nodja 4y ago
    You can finetune with it. If you want a more generic framework you can use hivemind[1] which is what petals uses, but you'll have to create your own community for whatever model you're trying to train. https://github.com/learning-at-home/hivemind https://github.com/learning-at-home/hivemind
  - nl 4y ago
    Obviously fine-tuning is just a special (easier) case of training. They are working on complete training of large models too: https://github.com/yandex-research/swarm https://github.com/yandex-research/swarm
rnosov 4y ago
The problem here is that most people just don't have suitable hardware. Ideally, you'd want to load the entire model into a GPU and most consumer grade GPUs just don't have nowhere near enough video memory. You'd need to have something like A100 80GB GPU to be able to run a node in the potential blockchain. You can buy one of these cards for about 15k USD. Admittedly, that's not that too far off from the price of a modern bitcoin ASIC miner but still a healthy chunk of change. And if you try to split the model across several GPUs then you'll have an issue of bandwidth as model parts would need to talk to each other (on the order of terabyte/second). At the moment, the only realistic way to contribute is just to provide feedback data for the RLHF training.

albertzeyer 4y ago

There is the Open Assistant project: https://github.com/LAION-AI/Open-Assistant https://github.com/LAION-AI/Open-Assistant There is also EleutherAI (https://www.eleuther.ai/about/ https://www.eleuther.ai/about/) with GPT-NeoX (https://github.com/EleutherAI/gpt-neox https://github.com/EleutherAI/gpt-neox).

moffkalast 4y ago

Just make sure it's written in Rust, uses a Sveltekit frontend and <some other buzzwords I can't remember right now>.

wg0 4y ago
And SQLite as local cache with CRDTs enabled whereas everything else from text search to queuing on PostgreSQL?

nodja 4y ago

https://petals.ml/ https://petals.ml/

Miraste 4y ago
Petals doesn't train new models, it only runs BLOOM in a distributed way.
- nodja 4y ago
  You can finetune with it. If you want a more generic framework you can use hivemind[1] which is what petals uses, but you'll have to create your own community for whatever model you're trying to train. https://github.com/learning-at-home/hivemind https://github.com/learning-at-home/hivemind
- nl 4y ago
  Obviously fine-tuning is just a special (easier) case of training. They are working on complete training of large models too: https://github.com/yandex-research/swarm https://github.com/yandex-research/swarm

rnosov 4y ago

The problem here is that most people just don't have suitable hardware. Ideally, you'd want to load the entire model into a GPU and most consumer grade GPUs just don't have nowhere near enough video memory. You'd need to have something like A100 80GB GPU to be able to run a node in the potential blockchain. You can buy one of these cards for about 15k USD. Admittedly, that's not that too far off from the price of a modern bitcoin ASIC miner but still a healthy chunk of change. And if you try to split the model across several GPUs then you'll have an issue of bandwidth as model parts would need to talk to each other (on the order of terabyte/second). At the moment, the only realistic way to contribute is just to provide feedback data for the RLHF training.