6 ms·

Why hasn't the community created a distributed Folding@home style GPT4/LLM software, where anyone can ask a question, and all participants machine's are contrib

Why hasn't the community created a distributed Folding@home style GPT4/LLM software, where anyone can ask a question, and all participants machine's are contributing to the computation of the answer? It seems like the ideal way for the open source community. Is it not possible for some reason with the way LLMs work and compute? Or has it simply just not been done yet?

simonw 3y ago
That exists: https://github.com/bigscience-workshop/petals https://github.com/bigscience-workshop/petals
brookst 3y ago
There's a bunch of stuff on this area. See https://www.fedai.org/ https://www.fedai.org/ for one. I also remember something more directly analogous to @home, but I'm having a hard time finding it. Edit: Petals! Thank you sibling commenter!
maffydub 3y ago
Interesting thought! I think the problem is that it's hard to parallelize this effectively across a network with Internet-scale latency - individual matrix multiplications parallelize very well, but there would need to be coordination of each result, which would be much slower than just doing the computation locally. In the case of Folding@home, evaluating possible folds could be done completely in parallel and only need to be coordinated on discovery of a plausible fold (which is rare), so distributing over a high-latency network is still beneficial.
shagie 3y ago
Distributing very large matrix multiplications doesn't scale across networks well. While this is a MINST classifier (and can be run in the browser) you can get an idea of the math behind it. https://www.3blue1brown.com/lessons/gradient-descent https://www.3blue1brown.com/lessons/gradient-descent and https://www.3blue1brown.com/lessons/neural-network-analysis https://www.3blue1brown.com/lessons/neural-network-analysis When dealing with a LLM, it's being run again and again - token by token. Pick the best next token, append it to the input, run it again. If you want to generate 100 tokens (rather small amount of data when compared to much of the GPT-4 conversations), that means running it 100 times. The network traffic makes the entire system much slower than running it all in one spot. Consider also the "are you the input to everyone?" and the privacy implications of that.
sounds 3y ago
The model can run across multiple machines but it's so slow that way that nobody bothered to ship that.

simonw 3y ago

That exists: https://github.com/bigscience-workshop/petals https://github.com/bigscience-workshop/petals

brookst 3y ago

There's a bunch of stuff on this area. See https://www.fedai.org/ https://www.fedai.org/ for one. I also remember something more directly analogous to @home, but I'm having a hard time finding it. Edit: Petals! Thank you sibling commenter!

maffydub 3y ago

Interesting thought! I think the problem is that it's hard to parallelize this effectively across a network with Internet-scale latency - individual matrix multiplications parallelize very well, but there would need to be coordination of each result, which would be much slower than just doing the computation locally. In the case of Folding@home, evaluating possible folds could be done completely in parallel and only need to be coordinated on discovery of a plausible fold (which is rare), so distributing over a high-latency network is still beneficial.

shagie 3y ago

Distributing very large matrix multiplications doesn't scale across networks well. While this is a MINST classifier (and can be run in the browser) you can get an idea of the math behind it. https://www.3blue1brown.com/lessons/gradient-descent https://www.3blue1brown.com/lessons/gradient-descent and https://www.3blue1brown.com/lessons/neural-network-analysis https://www.3blue1brown.com/lessons/neural-network-analysis When dealing with a LLM, it's being run again and again - token by token. Pick the best next token, append it to the input, run it again. If you want to generate 100 tokens (rather small amount of data when compared to much of the GPT-4 conversations), that means running it 100 times. The network traffic makes the entire system much slower than running it all in one spot. Consider also the "are you the input to everyone?" and the privacy implications of that.

sounds 3y ago

The model can run across multiple machines but it's so slow that way that nobody bothered to ship that.