4 ms·
> it is frustrating to be blocked from doing a lot of research in this space due to computational Do we need a SETI@home-like project to distribute the traini
by naasking 4y ago
> it is frustrating to be blocked from doing a lot of research in this space due to computational
Do we need a SETI@home-like project to distribute the
training computation across many volunteers so we can all benefit from the trained model?
- andai 4y agoI read about something a few weeks ago which does just this! Does anyone know what it's called?
- lucidrains 4y agoyou are probably thinking of https://arxiv.org/abs/2207.03481 https://arxiv.org/abs/2207.03481 for inference, there is https://github.com/bigscience-workshop/petals https://github.com/bigscience-workshop/petals however, both are only in the research phase. start tinkering!
- 8f2ab37a-ed6c 4y agoThat's brilliant, I would love to spare compute cycles and network on my devices for this if there's an open source LLM on the other side that I can use in my own projects, or commercially. Doesn't feel like there's much competition for ChatGPT at this point otherwise, which can't be good.
- davely 4y agoOn the generative image side of the equation, you can do the same thing with Stable Diffusion[1], thanks to a handy open source distributed computing project called Stable Horde[2]. LAION has started using Stable Horde for aesthetics training to back feed into and improve their datasets for future models[3]. I think one can foresee the same thing eventually happening with LLMs. Full disclosure: I made ArtBot, which is referenced in both the PC World article and the LAION blog post. [1] https://www.pcworld.com/article/1431633/meet-stable-horde-the-setihome-of-ai-art.html https://www.pcworld.com/article/1431633/meet-stable-horde-th... [2] https://stablehorde.net/ https://stablehorde.net/ [3] https://laion.ai/blog/laion-stable-horde/ https://laion.ai/blog/laion-stable-horde/
- vineyardmike 4y ago> Doesn't feel like there's much competition for ChatGPT at this point otherwise, which can't be good. Facebook open sourced their LLM, called OPT [1]. There's not much else, and OPT isn't exactly easy to run (requires like 8 GPUs). I'm not an expect, so I don't know why some models, like the graphics generation we've seen, are able to fit on phones, while LLM require $500k worth of GPUs to run. Hopefully this is the first step to changing that. [1] https://ai.facebook.com/blog/democratizing-access-to-large-scale-language-models-with-opt-175b/ https://ai.facebook.com/blog/democratizing-access-to-large-s...
- ec109685 4y agoAnother idea is to dedicate cpu cycles to something else that is easier to distribute, and then use the proceeds for massive amounts of gpu for academic use. Crypto is an example.
- slim 4y agothis would be very wasteful
- ec109685 4y agoSo is trying to distribute training across nodes compared to what can be done inside a data center.
- jxf 4y agoThis creates indirection costs and counterparty risks that don't appear in the original solution.
- ec109685 4y agoThere is also indirection cost by taking something that is optimized to run on GPU’s within the data center and distributing that to individual PCs.
- ikekkdcjkfke 4y agoYeah man, and youvget access to the model as payment for donati g cycles
- realce 4y agoHyperion
- zone411 4y agoLong story short, training requires intensive device-to-device communication. Distributed training is possible in theory but so inefficient that it's not worth it. Here is a new paper that looks to be the most promising approach yet: https://arxiv.org/abs/2301.11913 https://arxiv.org/abs/2301.11913
- nylonstrung 4y agoWould have to be federated learning to work I think
- sillysaurusx 4y agoIt doesn’t, actually. The model weights can be periodically averaged with each other. No need for synchronous gradient broadcasts. Why people aren’t doing this has always been a mystery to me. Relevant: https://battle.shawwn.com/swarm-training-v01a.pdf https://battle.shawwn.com/swarm-training-v01a.pdf
- telotortium 4y agoYou linked a paper with no results and no conclusion. Perhaps you meant to link a different paper?
- sillysaurusx 4y agoI never finished it.
- hackernewds 4y agoso it is unproven? what is the value of it?
- sillysaurusx 4y agoIt’s how we trained roughly 40 GPT 1.5B models. The technique works; it’s up to you to try it out.
- VadimPR 4y agoThat already exists - https://github.com/bigscience-workshop/petals https://github.com/bigscience-workshop/petals
- mdorazio 4y agoI've seen Petals mentioned several times before and I don't think it's the same thing. Correct me if I'm wrong, but it seems Petals is for running distributed inference and fine-tuning of an existing model. What the above poster and I really want to see is distributed training of a new model across a network. Much like I was able to choose to donate CPU cycles to a wide variety of BOINC-based projects, I want to be able to donate GPU cycles to anyone with a crazy idea for a new ML model - text, image, finance, audio, etc.
- Iv 4y agoHell it could even be the proof of work for a usable crypto-currency. "Prove that you lowered the error rate compared to SOTA and earn 50 ponzicoins!"
- qudat 4y agoThe labelled data seems more of a blocker than anything else. As far as I'm aware, the actually NN running the models are relatively simple, it's the human labor involved in gathering, cleaning, and labeling data for training that is the most resource intensive.
- naasking 4y agoThe data is valuable yes, but training a model still requires millions of dollars worth of compute. That's a perfect cost to distribute among volunteers if it could be done.