4 ms·

>If their cost in the cloud was about $1 per A100 hour, the training costs for this run alone would be about $63 million. If someone legitimate put together a

>If their cost in the cloud was about $1 per A100 hour, the training costs for this run alone would be about $63 million. If someone legitimate put together a crowd funding effort, I would donate a non-insignificant amount to train an open model. Has it been tried before?

asynchronous 3y ago
Not yet, heard tale of several people having the same idea to train an open model though through either crowdfunding or some wizardry with crowdsourcing GPUs. $65 million sounds pretty high though.
- fredoliveira 3y ago
  Considering an effort to buy a copy of the constitution raised almost $47M, I wouldn't be so sure. [^1] Worth noting, though, that it isn't just the computing budget that's missing here - it is also (and perhaps even more importantly) the high quality data to actually train the model. [^1]: https://en.wikipedia.org/wiki/ConstitutionDAO https://en.wikipedia.org/wiki/ConstitutionDAO
holoduke 3y ago
Some kind of SETI project, but for training a high number parameter llm would be awesome.
- drexlspivey 3y ago
  How many people have A100s at home?
  - TheRealPomax 3y ago
    The comparison to SETI@home[1] is that you just need the same mount of total processing power, not the same supercomputer setup. People don't need to own A100s, they just need to be willing to be part of a distributed supercomputer by running a background app that downloads chunks of data, processes them, and sends the result back. The utility comes from having enough people participate (which worked quite well for SETI@home, but helping find "signals from outer space" is a little bit more interesting than "helping train an LLM") [1] https://setiathome.berkeley.edu/ https://setiathome.berkeley.edu/
messe 3y ago
I too would be interesting in donating money toward this. Given that the price since the original training effort has already dropped to ~$20 million, and that (a) the fundraising will take time, and (b) improvements are being made every day with regard to resource usage, you could probably get away with aiming for a much lower number. Pulling a number out of my arse, I'd guess that training a comparable model will only cost $1-5 million in 12 months time, with the hardest part of doing so once you have the funds being acquiring the training data.

asynchronous 3y ago

Not yet, heard tale of several people having the same idea to train an open model though through either crowdfunding or some wizardry with crowdsourcing GPUs. $65 million sounds pretty high though.

fredoliveira 3y ago
Considering an effort to buy a copy of the constitution raised almost $47M, I wouldn't be so sure. [^1] Worth noting, though, that it isn't just the computing budget that's missing here - it is also (and perhaps even more importantly) the high quality data to actually train the model. [^1]: https://en.wikipedia.org/wiki/ConstitutionDAO https://en.wikipedia.org/wiki/ConstitutionDAO

holoduke 3y ago

Some kind of SETI project, but for training a high number parameter llm would be awesome.

drexlspivey 3y ago
How many people have A100s at home?
- TheRealPomax 3y ago
  The comparison to SETI@home[1] is that you just need the same mount of total processing power, not the same supercomputer setup. People don't need to own A100s, they just need to be willing to be part of a distributed supercomputer by running a background app that downloads chunks of data, processes them, and sends the result back. The utility comes from having enough people participate (which worked quite well for SETI@home, but helping find "signals from outer space" is a little bit more interesting than "helping train an LLM") [1] https://setiathome.berkeley.edu/ https://setiathome.berkeley.edu/

messe 3y ago

I too would be interesting in donating money toward this. Given that the price since the original training effort has already dropped to ~$20 million, and that (a) the fundraising will take time, and (b) improvements are being made every day with regard to resource usage, you could probably get away with aiming for a much lower number. Pulling a number out of my arse, I'd guess that training a comparable model will only cost $1-5 million in 12 months time, with the hardest part of doing so once you have the funds being acquiring the training data.