4 ms·

> If the technical preview is successful, our plan is to build a commercial version of GitHub Copilot in the future. This may be the first time that a propriet

> If the technical preview is successful, our plan is to build a commercial version of GitHub Copilot in the future. This may be the first time that a proprietary coding tool offers such a great value preposition that I am actually interested in trying it out and potentially even paying for it. It's also a bit concerning that this will probably be extremely hard, if not impossible, to create an FOSS version of this technology, just because of the immense amount of computing power, and by extension money, needed to create GPT3. I'm not that comfortable with the idea of a future where proprietary AI-based solutions and libraries (e.g. automatic testing libraries, which have been mentioned here a few times) are so powerful that I'll be forced to use them if I don't want to waste my time.

MathYouF 5y ago
Says the person who likely owns a washing machine, sink connected to plumbing, microwave, stove, lighters, clothes made with a sewing machine, ect. ect. GPT-3 will take way less time to make a good substitute that costs the power of compute than other historical time saving technologies. Unlike other historic technologies, they pretty much spell out exactly how to do it, and own no patents related to its creation. I have trouble seeing the downside.
- leotaku 5y ago
  I see your point, but aren't you making me into a bit of a straw man? When did I say that I was some open-access Luddite who won't use any technology that they can't build themselves. I just like the current state of programming, where I can productively build new and exciting things without having to rely on a lot of proprietary libraries or tools. Nothing more, nothing less.
  - MathYouF 5y ago
    Luddite: a person opposed to new technology or ways of working. Seems like you came up with the perfect word to describe your view on this. :P I defintiely do think there is reason to worry about the future you're imagining happening, but as someone who has read through the papers on gpt, this is in no way going to end up being an exclusive proprietary API, so it'll very soon have open source and likely free or at compute cost options. And well, if it doesn't make us better, no one will use it. If it does, you'll have to adjust to remain competitive, just like millions of professionals for going back millenia who encounter new technology in their chose and vocation. Ultimately, if it makes us better, and doesn't enslave us to the whims of a monopolistic holder (which the point of my post was to conjecture that it likely wouldn't) then it'll probably (though not defintiely) be better for us long term. Hopefully it doesn't erode programmers abilities like spell check seems to have eroded people's spelling abilities but I fear that's a likely side effect.
    - leotaku 5y ago
      Fair enough, it seems like we mostly agree on how technology generally affects the world. I can't say I currently possess the domain knowledge to confirm your views on GPT, so I'll just have to hope you are right.
0-_-0 5y ago
Well, it should be possible to crowdsource training a FOSS version, right? There should be a SETI-at-home for training neural networks. I would donate some GPU power for sure. SETI@home achieved 50 times the computing power of the world's largest supercomputer [0], so it might actually be the only way to train the future GPT4 or GPT5. [0]: https://en.wikipedia.org/wiki/SETI@home#Statistics https://en.wikipedia.org/wiki/SETI@home#Statistics
- hgial 5y ago
  At the moment crowd-sourcing ML training doesn't work very well because updates to the model's weights have to be shared between all of the nodes in the computation periodically. Giant models tend to be trained on clusters with very fast interconnects for this reason.
- devetec 5y ago
  Distributed GPT training doesn't really work (sadly). Maybe someone will get a good solution to that, though.
devetec 5y ago
There is a FOSS version of GPT-3 being worked on by EleutherAI, but they are only at 6 billion parameters so far, while the largest model of GPT-3 is 175 billion (more parameters is better (usually)). EleutherAI is getting more compute from CoreWeave to actually train the GPT-3-like FOSS model, so that's something to look forward to eventually :). Github Copilot uses Codex though, which seems to be a GPT model trained on just code by OpenAI. It wouldn't be too hard to train a FOSS version of Codex on the open-source code of Github and other sources.
wraptile 5y ago
it is absurdly ironic that OpenAi is turning out to be one of the biggest opennes perpetrators in the near-future/present.

MathYouF 5y ago

Says the person who likely owns a washing machine, sink connected to plumbing, microwave, stove, lighters, clothes made with a sewing machine, ect. ect. GPT-3 will take way less time to make a good substitute that costs the power of compute than other historical time saving technologies. Unlike other historic technologies, they pretty much spell out exactly how to do it, and own no patents related to its creation. I have trouble seeing the downside.

leotaku 5y ago
I see your point, but aren't you making me into a bit of a straw man? When did I say that I was some open-access Luddite who won't use any technology that they can't build themselves. I just like the current state of programming, where I can productively build new and exciting things without having to rely on a lot of proprietary libraries or tools. Nothing more, nothing less.
- MathYouF 5y ago
  Luddite: a person opposed to new technology or ways of working. Seems like you came up with the perfect word to describe your view on this. :P I defintiely do think there is reason to worry about the future you're imagining happening, but as someone who has read through the papers on gpt, this is in no way going to end up being an exclusive proprietary API, so it'll very soon have open source and likely free or at compute cost options. And well, if it doesn't make us better, no one will use it. If it does, you'll have to adjust to remain competitive, just like millions of professionals for going back millenia who encounter new technology in their chose and vocation. Ultimately, if it makes us better, and doesn't enslave us to the whims of a monopolistic holder (which the point of my post was to conjecture that it likely wouldn't) then it'll probably (though not defintiely) be better for us long term. Hopefully it doesn't erode programmers abilities like spell check seems to have eroded people's spelling abilities but I fear that's a likely side effect.
  - leotaku 5y ago
    Fair enough, it seems like we mostly agree on how technology generally affects the world. I can't say I currently possess the domain knowledge to confirm your views on GPT, so I'll just have to hope you are right.

0-_-0 5y ago

Well, it should be possible to crowdsource training a FOSS version, right? There should be a SETI-at-home for training neural networks. I would donate some GPU power for sure. SETI@home achieved 50 times the computing power of the world's largest supercomputer [0], so it might actually be the only way to train the future GPT4 or GPT5. [0]: https://en.wikipedia.org/wiki/SETI@home#Statistics https://en.wikipedia.org/wiki/SETI@home#Statistics

hgial 5y ago
At the moment crowd-sourcing ML training doesn't work very well because updates to the model's weights have to be shared between all of the nodes in the computation periodically. Giant models tend to be trained on clusters with very fast interconnects for this reason.
devetec 5y ago
Distributed GPT training doesn't really work (sadly). Maybe someone will get a good solution to that, though.

devetec 5y ago

There is a FOSS version of GPT-3 being worked on by EleutherAI, but they are only at 6 billion parameters so far, while the largest model of GPT-3 is 175 billion (more parameters is better (usually)). EleutherAI is getting more compute from CoreWeave to actually train the GPT-3-like FOSS model, so that's something to look forward to eventually :). Github Copilot uses Codex though, which seems to be a GPT model trained on just code by OpenAI. It wouldn't be too hard to train a FOSS version of Codex on the open-source code of Github and other sources.

wraptile 5y ago

it is absurdly ironic that OpenAi is turning out to be one of the biggest opennes perpetrators in the near-future/present.