3 ms·

For me another question is crucial: is it open (i.e. is there an open source reference implementation) or closed like the so-called "OpenAI" products/services?

mkolodny 6y ago
You can find the code for the paper here: https://github.com/timoschick/pet https://github.com/timoschick/pet
emteycz 6y ago
Can you please explain to me (I don't understand it) why do you think GPT-3 is closed? Yes, they won't share the trained model, but they're sharing the research here[0][1] so you can reproduce easily, aren't they? As I understand it now, it's very fair - training the model is a separate thing from doing (and sharing) the research, is very costly, and would not happen if they were forced to open that too - I also don't understand why should they be. [0] https://arxiv.org/abs/2005.14165 https://arxiv.org/abs/2005.14165 [1] https://github.com/openai/gpt-3 https://github.com/openai/gpt-3
- dj_mc_merlin 6y ago
  > so you can reproduce easily That's like saying you can look at the Eiffel tower and it's schematics so what's so hard about getting a spare million dollars and building it.
  - emteycz 6y ago
    Well, we others can buy a ticket and go see it. Same with GPT-3.
- fxtentacle 6y ago
  Yes they tell you how to do it. But training a model of this size requires you to use thousands of GPUs, or wait forever. That will sum up to millions of $$$ in rental and electricity costs.
  - emteycz 6y ago
    That seems like the reason why won't they share the trained model. Somebody needs to pay for that, how would they fund it if they just shared it? As far as my thinking goes, they are open more than enough. Thank you for your input!
    - fxtentacle 6y ago
      They claim to be a well funded nonprofit.
      - emteycz 6y ago
        Yeah, and that nonprofit probably wouldn't really be doing its mission it got funding for if they were handing out computing, instead of research
        fxtentacle 6y ago
        Well, they are withholding the information needed to verify their research. That is generally frowned upon in science. Plus, they are called OpenAI but producing a closed source product...
        emteycz 6y ago
        It's not a product, it all boils down to raw computing work. Their mission is to do open research, not provide computing time that they paid for for free. Similarly, Red Hat does not have to provide free cloud computing because they also contribute to Linux and use it.
  - dvfjsdhgfv 6y ago
    Looks like a great opportunity for someone to step in and organize a project aimed at recreating that. I for one would be more than glad to donate my GPUs' time for something that would be useful for all humanity, for free.
- luma 6y ago
  I submitted a request for access to the GPT-3 API a couple months ago and still haven't been approved. I don't find that to be very open at all.
  - emteycz 6y ago
    The API does not have to be open, the research does. This is like saying cloud services should be free and open for all because Linux is open.
    - emteycz 6y ago
      To expand on that point, they're basically selling pre-executed computing to you. Resolving the query itself is nothing.
    - dvfjsdhgfv 6y ago
      For quite some time maddog has been promoting Subutai, a kind of P2P cloud (https://subutai.io/ https://subutai.io/). Voluntary computing is an old concept, even BOINC is almost 20 years old. Peertube already reached the stage where it's actually usable and people can watch the movies smoothly. So it's not unimaginable that people who care organize somehow creating a platform where you could use a GPT-3 platform by contributing your GPU time in exchange.
- dynamite-ready 6y ago
  This is a very article on the subject of GPT3's 'market position' - https://bdtechtalks.com/2020/09/21/gpt-3-economy-business-model/ https://bdtechtalks.com/2020/09/21/gpt-3-economy-business-mo...

mkolodny 6y ago

You can find the code for the paper here: https://github.com/timoschick/pet https://github.com/timoschick/pet

emteycz 6y ago

Can you please explain to me (I don't understand it) why do you think GPT-3 is closed? Yes, they won't share the trained model, but they're sharing the research here[0][1] so you can reproduce easily, aren't they? As I understand it now, it's very fair - training the model is a separate thing from doing (and sharing) the research, is very costly, and would not happen if they were forced to open that too - I also don't understand why should they be. [0] https://arxiv.org/abs/2005.14165 https://arxiv.org/abs/2005.14165 [1] https://github.com/openai/gpt-3 https://github.com/openai/gpt-3

dj_mc_merlin 6y ago
> so you can reproduce easily That's like saying you can look at the Eiffel tower and it's schematics so what's so hard about getting a spare million dollars and building it.
- emteycz 6y ago
  Well, we others can buy a ticket and go see it. Same with GPT-3.
fxtentacle 6y ago
Yes they tell you how to do it. But training a model of this size requires you to use thousands of GPUs, or wait forever. That will sum up to millions of $$$ in rental and electricity costs.
- emteycz 6y ago
  That seems like the reason why won't they share the trained model. Somebody needs to pay for that, how would they fund it if they just shared it? As far as my thinking goes, they are open more than enough. Thank you for your input!
  - fxtentacle 6y ago
    They claim to be a well funded nonprofit.
    - emteycz 6y ago
      Yeah, and that nonprofit probably wouldn't really be doing its mission it got funding for if they were handing out computing, instead of research
      - fxtentacle 6y ago
        Well, they are withholding the information needed to verify their research. That is generally frowned upon in science. Plus, they are called OpenAI but producing a closed source product...
        emteycz 6y ago
        It's not a product, it all boils down to raw computing work. Their mission is to do open research, not provide computing time that they paid for for free. Similarly, Red Hat does not have to provide free cloud computing because they also contribute to Linux and use it.
- dvfjsdhgfv 6y ago
  Looks like a great opportunity for someone to step in and organize a project aimed at recreating that. I for one would be more than glad to donate my GPUs' time for something that would be useful for all humanity, for free.
luma 6y ago
I submitted a request for access to the GPT-3 API a couple months ago and still haven't been approved. I don't find that to be very open at all.
- emteycz 6y ago
  The API does not have to be open, the research does. This is like saying cloud services should be free and open for all because Linux is open.
  - emteycz 6y ago
    To expand on that point, they're basically selling pre-executed computing to you. Resolving the query itself is nothing.
  - dvfjsdhgfv 6y ago
    For quite some time maddog has been promoting Subutai, a kind of P2P cloud (https://subutai.io/ https://subutai.io/). Voluntary computing is an old concept, even BOINC is almost 20 years old. Peertube already reached the stage where it's actually usable and people can watch the movies smoothly. So it's not unimaginable that people who care organize somehow creating a platform where you could use a GPT-3 platform by contributing your GPU time in exchange.
dynamite-ready 6y ago
This is a very article on the subject of GPT3's 'market position' - https://bdtechtalks.com/2020/09/21/gpt-3-economy-business-model/ https://bdtechtalks.com/2020/09/21/gpt-3-economy-business-mo...