4 ms·

GPT Code Clippy, the Open Source Version of GitHub Copilot

115 points by elashri 5y ago

terafo 5y ago
I wish there was some kind of extension to run this thing locally on GPU, Deep TabNine is 2 years old now and still nothing tries to compete with it despite it using smallest possible gpt2 with cpu inference.
- bredren 5y ago
  I’m also surprised at the lack of competition. I’ve been using Tabnine since last summer. Tabnine was so compelling I posted here not long after using it that AI-based code completion was being slept on, because it was obviously so much better than what the IDE could do. [1] I pointed out that Microsoft was uniquely positioned to offer superior code completion services due to its fire-hose access to new code on GitHub. Interestingly, the top reply from ptx was their “hope we'll also see a strong counter-movement of simple tools for independent developers.” Which is just what Code Clippy seems to attempt. I continue to think AI-based code completion and AI assisted code quality recommendation services will be the differentiator as so much of IDE features are commoditized. The big missing party in my view is Apple with Xcode. Not only do I think Xcode needs it’s own local-capable AI code completion, I think Apple needs a semi-social GitHub-like repository service of its own to help feed this AI. [1] https://news.ycombinator.com/item?id=25074393 https://news.ycombinator.com/item?id=25074393
  - IceWreck 5y ago
    > I think Apple needs a semi-social GitHub-like repository service of its own to help feed this AI. Why? Can't they lift code off github, gitlab, etc ? I don't remember giving Microsoft permission to use my code to train Copilot. They did it because it was public. Apple can always clone all public repos and do it themselves.
    - bredren 5y ago
      I think in order to provide a cohesive, cloud-assisted developer experience they must integrate all the way to the code. Also, there will likely be Apple service secrets stored alongside that, and Apple will want to be the ones responsible for protecting those. For example, GitHub Actions like workflows on top of iOS projects
moyix 5y ago
Awesome to see open and independent reproductions of Codex/Copilot being worked on! It seems like this must be in fairly early stages? The HumanEval numbers in the README seem to indicate the pass rate with one sample (pass@1) on the benchmark is at best 0.06%, whereas the Codex paper reports up to 28% (for the Codex-12B model). I'm actually kind of surprised by this, since even the very small Codex models (e.g., Codex-12M) manage 2% on that benchmark. I also don't see a lot of recent activity – anyone know if the project is still alive?
omgitsabird 5y ago
Is copilot just a search engine over function and variable names with some filters and transformers (like a transcompiler)? I just got into the preview today, and it seems like that is what it is to me.
- moyix 5y ago
  It's not. It's basically the same model as GPT-3 (though smaller), and generally speaking the vast majority of its outputs should not be anything in its training set. The big exception is code that showed up many times in the training data (like the famous inverse square root function).
- paavohtl 5y ago
  It is based on a machine learning model. I've been using it for a couple of months now and the suggestions are generally speaking surprisingly good. It has been able to usefully complete very obscure things, like parts of a PEG grammar for parsing Quake-C. I'm not using it to generate implementations of algorithms from scratch (because it just doesn't have enough data to do that properly), but rather as a fancy extension of regular code completion.
  - hutzlibu 5y ago
    I imagine being offered complex code completion can lead to just accept it at times, without understanding it fully and therefore lots of potential for hidden bugs. Did you notice anything like this?
    - paavohtl 5y ago
      It is quite rare, but happens sometimes. You basically have to explicitly ask for a complex completion by writing a specification in a comment. Normally it tends to just complete a single statement or other syntax element at a time, which are usually easy to read and pretty close to correct.
tsavo_knott 5y ago
code.pieces.app is utilizing some of this to do hyper local suggestions & auto completing. I was pretty bummed to see Copilot being closed source but trained on open source so big +1 on all the efforts here. Small companies like ours really could utilize these amazing technologies and they need to be democratized.
- NmAmDa 5y ago
  The app seems very interesting, I only wish that you can publish the extension to open-vsx.org so that people who are using VSCodium and Code-OSS in general can use it.

115 points by elashri 5y ago

I wish there was some kind of extension to run this thing locally on GPU, Deep TabNine is 2 years old now and still nothing tries to compete with it despite it using smallest possible gpt2 with cpu inference.

bredren 5y ago
I’m also surprised at the lack of competition. I’ve been using Tabnine since last summer. Tabnine was so compelling I posted here not long after using it that AI-based code completion was being slept on, because it was obviously so much better than what the IDE could do. [1] I pointed out that Microsoft was uniquely positioned to offer superior code completion services due to its fire-hose access to new code on GitHub. Interestingly, the top reply from ptx was their “hope we'll also see a strong counter-movement of simple tools for independent developers.” Which is just what Code Clippy seems to attempt. I continue to think AI-based code completion and AI assisted code quality recommendation services will be the differentiator as so much of IDE features are commoditized. The big missing party in my view is Apple with Xcode. Not only do I think Xcode needs it’s own local-capable AI code completion, I think Apple needs a semi-social GitHub-like repository service of its own to help feed this AI. [1] https://news.ycombinator.com/item?id=25074393 https://news.ycombinator.com/item?id=25074393
- IceWreck 5y ago
  > I think Apple needs a semi-social GitHub-like repository service of its own to help feed this AI. Why? Can't they lift code off github, gitlab, etc ? I don't remember giving Microsoft permission to use my code to train Copilot. They did it because it was public. Apple can always clone all public repos and do it themselves.
  - bredren 5y ago
    I think in order to provide a cohesive, cloud-assisted developer experience they must integrate all the way to the code. Also, there will likely be Apple service secrets stored alongside that, and Apple will want to be the ones responsible for protecting those. For example, GitHub Actions like workflows on top of iOS projects

moyix 5y ago

Awesome to see open and independent reproductions of Codex/Copilot being worked on! It seems like this must be in fairly early stages? The HumanEval numbers in the README seem to indicate the pass rate with one sample (pass@1) on the benchmark is at best 0.06%, whereas the Codex paper reports up to 28% (for the Codex-12B model). I'm actually kind of surprised by this, since even the very small Codex models (e.g., Codex-12M) manage 2% on that benchmark. I also don't see a lot of recent activity – anyone know if the project is still alive?

omgitsabird 5y ago

Is copilot just a search engine over function and variable names with some filters and transformers (like a transcompiler)? I just got into the preview today, and it seems like that is what it is to me.

moyix 5y ago
It's not. It's basically the same model as GPT-3 (though smaller), and generally speaking the vast majority of its outputs should not be anything in its training set. The big exception is code that showed up many times in the training data (like the famous inverse square root function).
paavohtl 5y ago
It is based on a machine learning model. I've been using it for a couple of months now and the suggestions are generally speaking surprisingly good. It has been able to usefully complete very obscure things, like parts of a PEG grammar for parsing Quake-C. I'm not using it to generate implementations of algorithms from scratch (because it just doesn't have enough data to do that properly), but rather as a fancy extension of regular code completion.
- hutzlibu 5y ago
  I imagine being offered complex code completion can lead to just accept it at times, without understanding it fully and therefore lots of potential for hidden bugs. Did you notice anything like this?
  - paavohtl 5y ago
    It is quite rare, but happens sometimes. You basically have to explicitly ask for a complex completion by writing a specification in a comment. Normally it tends to just complete a single statement or other syntax element at a time, which are usually easy to read and pretty close to correct.

tsavo_knott 5y ago

code.pieces.app is utilizing some of this to do hyper local suggestions & auto completing. I was pretty bummed to see Copilot being closed source but trained on open source so big +1 on all the efforts here. Small companies like ours really could utilize these amazing technologies and they need to be democratized.

NmAmDa 5y ago
The app seems very interesting, I only wish that you can publish the extension to open-vsx.org so that people who are using VSCodium and Code-OSS in general can use it.