4 ms·

Honestly, I think the main problem is, that training large models like this, or GPT-any, or anything like that costs a lot of compute power - which costs a lot

by sva_ 5y ago

Honestly, I think the main problem is, that training large models like this, or GPT-any, or anything like that costs a lot of compute power - which costs a lot of money. So if you're just investing a lot of compute but give everything away for free, that isn't a strategy which can be maintained. So what we really need, are ways to train models in a distributed fashion. Their initial goal was to 'democratize AI'. But democratizing these models doesn't just mean giving everyone access to your trained parameters, it also means that everyone should be able to spend an effort on improving it. I think a good solution for this would be huge.

emilsedgh 5y ago
So when they formed OpenAI and thought "we are gonna hire the most brilliant scientists in the field" they forgot to account for hardware costs? At this point it's more like doublespeak.
- moffkalast 5y ago
  They just announced they'll be rebranding as OpenYourWallet
  - texasbigdata 5y ago
    Do they have a revenue model? I thought it was a non-profit / public benefit company.
leereeves 5y ago
> So what we really need, are ways to train models in a distributed fashion. How is the replication effort for this project handling that? They have over 600 people on their Discord right now, do they have some way for them to cooperate on training a single model?
- sva_ 5y ago
  As far as I can see, No.
  - astrange 5y ago
    GPT-J and GPT-Neo are previous open source reproductions of GPT, but distributed training is overrated. There’s way too much communications overhead. I’m not sure any of the various @home projects have ever accomplished much for that matter. Instead they use donated time on a TPU or a GPU cloud provider. An exception is Leela Zero since you can realistically have it play chess at home.
    - airstrike 5y ago
      > Instead they use donated time on a TPU or a GPU cloud provider. So instead distribute the funding? And distributing running models too? Something something crowdfunding blockchain cryptocurrency Or alternatively https://www.microsoft.com/en-us/research/publication/decentralized-collaborative-ai-on-blockchain/ https://www.microsoft.com/en-us/research/publication/decentr...
    - kache_ 5y ago
      >donated time on a TPU or a GPU cloud provider hmmm... what about people who mine crypto on their hardware, and then that hardware goes into a pool of shared $ used to rent out a TPU
joshcryer 5y ago
The training set is a one time computation though, running the model still costs cycles and people have to pay for it (see sites like Hugging Face or GooseAI). What people are asking about is the code itself and the one time computed training set. They're not asking to run the trained model for free. Anyway, getting cycles isn't too hard, expect CoreWeave or another compute-centric company to hop on board in due course.
- moffkalast 5y ago
  > They're not asking to run the trained model for free. I am :D
gisely 5y ago
The could have chosen to work on smaller models and used their already substantial amount of funding to be truly open. Instead they chose a path that demonstrates the Open part in OpenAI was just marketing.

by sva_ 5y ago

emilsedgh 5y ago

So when they formed OpenAI and thought "we are gonna hire the most brilliant scientists in the field" they forgot to account for hardware costs? At this point it's more like doublespeak.

moffkalast 5y ago
They just announced they'll be rebranding as OpenYourWallet
- texasbigdata 5y ago
  Do they have a revenue model? I thought it was a non-profit / public benefit company.

leereeves 5y ago

> So what we really need, are ways to train models in a distributed fashion. How is the replication effort for this project handling that? They have over 600 people on their Discord right now, do they have some way for them to cooperate on training a single model?

sva_ 5y ago
As far as I can see, No.
- astrange 5y ago
  GPT-J and GPT-Neo are previous open source reproductions of GPT, but distributed training is overrated. There’s way too much communications overhead. I’m not sure any of the various @home projects have ever accomplished much for that matter. Instead they use donated time on a TPU or a GPU cloud provider. An exception is Leela Zero since you can realistically have it play chess at home.
  - airstrike 5y ago
    > Instead they use donated time on a TPU or a GPU cloud provider. So instead distribute the funding? And distributing running models too? Something something crowdfunding blockchain cryptocurrency Or alternatively https://www.microsoft.com/en-us/research/publication/decentralized-collaborative-ai-on-blockchain/ https://www.microsoft.com/en-us/research/publication/decentr...
  - kache_ 5y ago
    >donated time on a TPU or a GPU cloud provider hmmm... what about people who mine crypto on their hardware, and then that hardware goes into a pool of shared $ used to rent out a TPU

joshcryer 5y ago

The training set is a one time computation though, running the model still costs cycles and people have to pay for it (see sites like Hugging Face or GooseAI). What people are asking about is the code itself and the one time computed training set. They're not asking to run the trained model for free. Anyway, getting cycles isn't too hard, expect CoreWeave or another compute-centric company to hop on board in due course.

moffkalast 5y ago
> They're not asking to run the trained model for free. I am :D

gisely 5y ago

The could have chosen to work on smaller models and used their already substantial amount of funding to be truly open. Instead they chose a path that demonstrates the Open part in OpenAI was just marketing.