6 ms·

The VSCode GitLab extension now supports getting code completions from FauxPilot

190 points by moyix 4y ago

cercatrova 4y ago
Good to see the OSSification (no pun intended) of prorpietary VSCode parts such as Copilot and their extension server.
- Havoc 4y ago
  Extension server has a oss now?
  - commoner 4y ago
    Yes, it's called Open VSX and it's developed by the Eclipse Foundation. - Open VSX: https://open-vsx.org https://open-vsx.org - Source: https://github.com/eclipse/openvsx https://github.com/eclipse/openvsx VSCodium and Eclipse Theia use Open VSX by default. - VSCodium: https://github.com/VSCodium/vscodium#extensions-and-the-marketplace https://github.com/VSCodium/vscodium#extensions-and-the-mark... - Eclipse: https://www.eclipse.org/community/eclipse_newsletter/2020/march/1.php https://www.eclipse.org/community/eclipse_newsletter/2020/ma...
    - Havoc 4y ago
      Thank you!
    - nextaccountic 4y ago
      Open VSX allows anyone to upload an extension there with the same name and description from the VSCode marketplace, but silently change code or make new releases, maybe introducing misfeatures or malicious code. Users typically don't notice because they think they are installing "the same" extension from the VSCode marketplace. https://github.com/hediet/vscode-drawio/issues/141 https://github.com/hediet/vscode-drawio/issues/141 > Hi ;) > I never published a version v999.0! It seems like you are using the unofficial open vsx marketplace (where, apparently, anyone can upload anything). You can find an issue here in this repository about it. > Unfortunately, someone uploaded the extension in that version which blocks any further updates with that name. > For now I believe in Microsofts vision. I don't think a secondary marketplace is good for the community - It just causes confusions like this. > If you setup a github action that automatically publishes this extension to open vsx, please open a PR! ;) The established practice of having random individuals set up ad-hoc mirrors of VSCode extensions is a serious security issue. If Open VSX wants to mirror VSCode extensions, that's okay - but they should do so with an automated process that mirrors ALL extensions and do not allow for random people to silently change the code of an extension with no clear indication to people installing it. If, however, someone want to copy the code of an existing VSCode extension, change some things and upload it to Open VSX, that's super okay too (and in the spirit of open source), but please fork it and clearly indicate in the description that the extension is a fork, linking to the source code of the original extension. The currently situation is unacceptable.
      - nextaccountic 4y ago
        And I want to add that it's Microsoft that is in the wrong here. Their policy of only allowing the usage of their package repository if you are using their proprietary build of VSCode is absurd. It's as if npm disallowed the use of their repositories by yarn and pnpm. We shouldn't tolerate this behavior, specially not from a company that claims to "love open source". But, Open VSX could and should do more for people to verify the provenance of their packages. There are many ways to do this. Perhaps one way is to have two kinds of packages (readily apparent in the description): one automatically imported from the VSCode marketplace (and guaranteed to match the upstream package exactly), and another kind published specifically for Open VSX. Right now it seems to be a better security practice to simply ignore the VSCode marketplace terms of use and use it anyway on open source builds (either Code - OSS or VS Codium), instead of using Open VSX. And that's a shitty situation to be in.
syntaxing 4y ago
This is super awesome, too bad a A6000 cost $4500 or I would try this out myself.
- moyix 4y ago
  The smaller models run on smaller GPUs too! :) You can see how much VRAM is required for various models in the Documentation: https://github.com/moyix/fauxpilot/blob/main/documentation/server.md#setup https://github.com/moyix/fauxpilot/blob/main/documentation/s... And we have an initial GPU support matrix page here: https://github.com/moyix/fauxpilot/wiki/GPU-Support-Matrix https://github.com/moyix/fauxpilot/wiki/GPU-Support-Matrix A planned feature is to implement INT8 and INT4 support for the CodeGen models, which would let the models run with much less VRAM (~2x smaller for INT8 and ~4x for INT4) :)
  - Namidairo 4y ago
    > A planned feature is to implement INT8 and INT4 support for the CodeGen models, which would let the models run with much less VRAM (~2x smaller for INT8 and ~4x for INT4) :) That would certainly put the larger models within reach of most users. Part of my reluctance to deploy this myself was I'd have to either redline the VRAM on a 8GB card or choose the smallest model. Would this require a retrain of the models?
    - moyix 4y ago
      It shouldn't require retraining, nope. I believe for INT4 there is a small adapter layer added that needs to be trained, but it is small and wouldn't require much data or computation to do so. Will know more once we actually start implementing :)
  - daviding 4y ago
    The 7GB runs great on a 3080Ti. I am getting a lot of 'ValueError: Max tokens + prompt length' errors with larger files. Can this Gitlab client also replace the vocab.bpe and tokenizer.json config like Copilots? Thanks for your work on Fauxpilot, really enjoying playing with it.
john_cogs 4y ago
GitLab team member here. This work was produced by our AI Assist SEG (Single-Engineer Group)[0]. The engineer behind this feature recently uploaded a quick update about this work and other things they are working on to YouTube[1]. [0] - https://about.gitlab.com/handbook/engineering/incubation/ai-assist/ https://about.gitlab.com/handbook/engineering/incubation/ai-... [1] - https://www.youtube.com/watch?v=VYl0dg8xyeE https://www.youtube.com/watch?v=VYl0dg8xyeE
- remram 4y ago
  Why do you call this a group? Why don't you say "one of our engineers did this"? I read the linked article [1] and that seems to be the accurate situation: not a group of people including one engineer, not one engineer at a time on a rotation, but literally one person. In what way is that a group? [1]: https://about.gitlab.com/company/team/structure/#single-engineer-groups https://about.gitlab.com/company/team/structure/#single-engi...
  - john_cogs 4y ago
    Great question. I'm not entirely clear on the origin of the name and it would probably be hard for me to find the folks behind this decision on Friday evening/Saturday so I'll share my interpretation. At GitLab, we have a clearly defined organizational structure. Within that stucture, we have Product Groups[0] which are groups of people aligned around a specific category. The name "Single-Engineer Groups" reflects that this single engineer owns the category which they're focusing on. I'll be sure to surface your question to the leader of our Incubation Engineering org. Thanks. [0] - https://about.gitlab.com/company/team/structure/#product-groups https://about.gitlab.com/company/team/structure/#product-gro...
    - cercatrova 4y ago
      I imagine it's similar to working groups [0], but keeping the term the same when only one person is in the working group, hence single engineer (working) group. Basically, parse `group` as in `working group` as a single term of art, rather than literally meaning a colloquial group, ie 2 or more individuals. [0] https://en.wikipedia.org/wiki/Working_group https://en.wikipedia.org/wiki/Working_group
abirch 4y ago
This is so awesome!!! Your team rocks. Too bad it wasn't named after your cat: FelinePilot or CatPilot.
- arvonle 4y ago
  The Gitlab mascot is a raccoon dog, closely related to the true fox. FauxPilot.
- moyix 4y ago
  I will inform Andrei and Katya that they have a fan :) Although they already have a pretty high opinion of themselves!
sva_ 4y ago
From https://github.com/moyix/fauxpilot https://github.com/moyix/fauxpilot > if you have two NVIDIA RTX 3080 GPUs, you should be able to run the 6B model by putting half on each GPU. Thats quite excessive, but nice that it can be ran locally. > Note that the VRAM requirements listed by setup.sh are total -- if you have multiple GPUs, you can split the model across them. This might not work with the upcoming RTX 40x0 GPUs unless they have enough memory, as they abolished NVLink in that series. Edit: I'm now seeing there are smaller models too. Going to give it a try tomorrow.
- deleted 4y ago
  [deleted]
- woodson 4y ago
  You don’t need NVlink for that to work.
  - sva_ 4y ago
    Apologies for my ignorance, but don't you need NVLink to pool the memory of the cards together?
    - moyix 4y ago
      Nope. The way it works is FasterTransformer splits the model across the two GPUs and runs both halves in parallel. It periodically has to sync the results from each half, so it will go faster if you have a high-bandwidth link between the GPUs like NVLink, but it will work just fine even if they have to communicate over PCIe peer-to-peer or even communicating via the CPU.
doctoboggan 4y ago
I know there seem to be a few different models to choose from, does anyone have a resource that collects them and shows the strengths and weaknesses of the various options? I’d be particularly interested in models optimized for apple silicon. Also, how do they compare to copilot?
- Vetch 4y ago
  Unfortunately, Copilot is a lot more capable. Most important is that it works with many more languages out of the box, is continuously updated, has more mathematical plus scientific knowledge and is better at understanding your comments. As for which to use, you'd want to take the largest model that can you can fine-tune (not necessarily on your machine) and fits comfortably in your machine. The small models aren't as flexible and are basically just a cleverer autocomplete. Copilot is a lot smarter than that and smarter than all models I've tested up to 6B. It can often cleverly work out a lot of local patterns for what you're doing and this is true surprisingly often for novel languages, math, understanding graph notation, categorization policies and much, much more. I'd gone in hoping I could remove my dependence on Copilot but left disappointed. The small models up to 6B just don't compare. I don't have the resources to run the 13B model which is probably behind anyways due to not having trained as long on as much as copilot (which is continuously updated). I also suspect Copilot has a great deal of hand-coded hacks and caching tricks to improve the user experience.
  - moyix 4y ago
    Yep, I definitely agree that the 6B and below models are worse than Copilot. The 16B ones are pretty good! But IMO still undertrained compared to Copilot, and of course much less accessible (though see elsewhere in this comment section; I think INT8 and even INT4 should be doable – this won't help much with inference latency but it should help most people fit the 16B model locally). I have high hopes for the BigCode project too; they will get to take advantage of a lot of things that have been learned about how to train code models effectively. One last note – I think that many of the improvements to Copilot since its release are attributable to "prompt engineering" and client-side smarts about what context should be included; I'm not certain, but I can believe the underlying Codex model used hasn't changed much (if at all) since code-davinci-002.
dang 4y ago
Related: FauxPilot – an attempt to build a locally hosted version of GitHub Copilot - https://news.ycombinator.com/item?id=32327711 https://news.ycombinator.com/item?id=32327711 - Aug 2022 (79 comments)
daviding 4y ago
This seems to work ok on a 3080Ti, even under WSL2. Nice! For larger files I'm getting errors for maximum token + prompt length - is there an easy way to tweak these limits for the client? I think the Copilot client needed some overrides for Fauxpilot for this, so hoping Gitlab client has that too.
speedgoose 4y ago
Shouldn’t this be in a separate extension?
- avip3d 4y ago
  That would make sense... but them most people wouldn't install the base gitlab extension.
- eddyg 4y ago
  Came here to say this. I’d much rather see effort being put into other areas of the GitLab Workflow extension (like being able to approve / view approvals of MRs).
ddevault 4y ago
You'd think that GitLab would pause from chasing after GitHub's tail for at least long enough to question whether or not these features are ethical.

190 points by moyix 4y ago

cercatrova 4y ago

Good to see the OSSification (no pun intended) of prorpietary VSCode parts such as Copilot and their extension server.

Havoc 4y ago
Extension server has a oss now?
- commoner 4y ago
  Yes, it's called Open VSX and it's developed by the Eclipse Foundation. - Open VSX: https://open-vsx.org https://open-vsx.org - Source: https://github.com/eclipse/openvsx https://github.com/eclipse/openvsx VSCodium and Eclipse Theia use Open VSX by default. - VSCodium: https://github.com/VSCodium/vscodium#extensions-and-the-marketplace https://github.com/VSCodium/vscodium#extensions-and-the-mark... - Eclipse: https://www.eclipse.org/community/eclipse_newsletter/2020/march/1.php https://www.eclipse.org/community/eclipse_newsletter/2020/ma...
  - Havoc 4y ago
    Thank you!
  - nextaccountic 4y ago
    Open VSX allows anyone to upload an extension there with the same name and description from the VSCode marketplace, but silently change code or make new releases, maybe introducing misfeatures or malicious code. Users typically don't notice because they think they are installing "the same" extension from the VSCode marketplace. https://github.com/hediet/vscode-drawio/issues/141 https://github.com/hediet/vscode-drawio/issues/141 > Hi ;) > I never published a version v999.0! It seems like you are using the unofficial open vsx marketplace (where, apparently, anyone can upload anything). You can find an issue here in this repository about it. > Unfortunately, someone uploaded the extension in that version which blocks any further updates with that name. > For now I believe in Microsofts vision. I don't think a secondary marketplace is good for the community - It just causes confusions like this. > If you setup a github action that automatically publishes this extension to open vsx, please open a PR! ;) The established practice of having random individuals set up ad-hoc mirrors of VSCode extensions is a serious security issue. If Open VSX wants to mirror VSCode extensions, that's okay - but they should do so with an automated process that mirrors ALL extensions and do not allow for random people to silently change the code of an extension with no clear indication to people installing it. If, however, someone want to copy the code of an existing VSCode extension, change some things and upload it to Open VSX, that's super okay too (and in the spirit of open source), but please fork it and clearly indicate in the description that the extension is a fork, linking to the source code of the original extension. The currently situation is unacceptable.
    - nextaccountic 4y ago
      And I want to add that it's Microsoft that is in the wrong here. Their policy of only allowing the usage of their package repository if you are using their proprietary build of VSCode is absurd. It's as if npm disallowed the use of their repositories by yarn and pnpm. We shouldn't tolerate this behavior, specially not from a company that claims to "love open source". But, Open VSX could and should do more for people to verify the provenance of their packages. There are many ways to do this. Perhaps one way is to have two kinds of packages (readily apparent in the description): one automatically imported from the VSCode marketplace (and guaranteed to match the upstream package exactly), and another kind published specifically for Open VSX. Right now it seems to be a better security practice to simply ignore the VSCode marketplace terms of use and use it anyway on open source builds (either Code - OSS or VS Codium), instead of using Open VSX. And that's a shitty situation to be in.

syntaxing 4y ago

This is super awesome, too bad a A6000 cost $4500 or I would try this out myself.

moyix 4y ago
The smaller models run on smaller GPUs too! :) You can see how much VRAM is required for various models in the Documentation: https://github.com/moyix/fauxpilot/blob/main/documentation/server.md#setup https://github.com/moyix/fauxpilot/blob/main/documentation/s... And we have an initial GPU support matrix page here: https://github.com/moyix/fauxpilot/wiki/GPU-Support-Matrix https://github.com/moyix/fauxpilot/wiki/GPU-Support-Matrix A planned feature is to implement INT8 and INT4 support for the CodeGen models, which would let the models run with much less VRAM (~2x smaller for INT8 and ~4x for INT4) :)
- Namidairo 4y ago
  > A planned feature is to implement INT8 and INT4 support for the CodeGen models, which would let the models run with much less VRAM (~2x smaller for INT8 and ~4x for INT4) :) That would certainly put the larger models within reach of most users. Part of my reluctance to deploy this myself was I'd have to either redline the VRAM on a 8GB card or choose the smallest model. Would this require a retrain of the models?
  - moyix 4y ago
    It shouldn't require retraining, nope. I believe for INT4 there is a small adapter layer added that needs to be trained, but it is small and wouldn't require much data or computation to do so. Will know more once we actually start implementing :)
- daviding 4y ago
  The 7GB runs great on a 3080Ti. I am getting a lot of 'ValueError: Max tokens + prompt length' errors with larger files. Can this Gitlab client also replace the vocab.bpe and tokenizer.json config like Copilots? Thanks for your work on Fauxpilot, really enjoying playing with it.

john_cogs 4y ago

GitLab team member here. This work was produced by our AI Assist SEG (Single-Engineer Group)[0]. The engineer behind this feature recently uploaded a quick update about this work and other things they are working on to YouTube[1]. [0] - https://about.gitlab.com/handbook/engineering/incubation/ai-assist/ https://about.gitlab.com/handbook/engineering/incubation/ai-... [1] - https://www.youtube.com/watch?v=VYl0dg8xyeE https://www.youtube.com/watch?v=VYl0dg8xyeE

remram 4y ago
Why do you call this a group? Why don't you say "one of our engineers did this"? I read the linked article [1] and that seems to be the accurate situation: not a group of people including one engineer, not one engineer at a time on a rotation, but literally one person. In what way is that a group? [1]: https://about.gitlab.com/company/team/structure/#single-engineer-groups https://about.gitlab.com/company/team/structure/#single-engi...
- john_cogs 4y ago
  Great question. I'm not entirely clear on the origin of the name and it would probably be hard for me to find the folks behind this decision on Friday evening/Saturday so I'll share my interpretation. At GitLab, we have a clearly defined organizational structure. Within that stucture, we have Product Groups[0] which are groups of people aligned around a specific category. The name "Single-Engineer Groups" reflects that this single engineer owns the category which they're focusing on. I'll be sure to surface your question to the leader of our Incubation Engineering org. Thanks. [0] - https://about.gitlab.com/company/team/structure/#product-groups https://about.gitlab.com/company/team/structure/#product-gro...
  - cercatrova 4y ago
    I imagine it's similar to working groups [0], but keeping the term the same when only one person is in the working group, hence single engineer (working) group. Basically, parse `group` as in `working group` as a single term of art, rather than literally meaning a colloquial group, ie 2 or more individuals. [0] https://en.wikipedia.org/wiki/Working_group https://en.wikipedia.org/wiki/Working_group

abirch 4y ago

This is so awesome!!! Your team rocks. Too bad it wasn't named after your cat: FelinePilot or CatPilot.

arvonle 4y ago
The Gitlab mascot is a raccoon dog, closely related to the true fox. FauxPilot.
moyix 4y ago
I will inform Andrei and Katya that they have a fan :) Although they already have a pretty high opinion of themselves!

sva_ 4y ago

From https://github.com/moyix/fauxpilot https://github.com/moyix/fauxpilot > if you have two NVIDIA RTX 3080 GPUs, you should be able to run the 6B model by putting half on each GPU. Thats quite excessive, but nice that it can be ran locally. > Note that the VRAM requirements listed by setup.sh are total -- if you have multiple GPUs, you can split the model across them. This might not work with the upcoming RTX 40x0 GPUs unless they have enough memory, as they abolished NVLink in that series. Edit: I'm now seeing there are smaller models too. Going to give it a try tomorrow.

deleted 4y ago
[deleted]
woodson 4y ago
You don’t need NVlink for that to work.
- sva_ 4y ago
  Apologies for my ignorance, but don't you need NVLink to pool the memory of the cards together?
  - moyix 4y ago
    Nope. The way it works is FasterTransformer splits the model across the two GPUs and runs both halves in parallel. It periodically has to sync the results from each half, so it will go faster if you have a high-bandwidth link between the GPUs like NVLink, but it will work just fine even if they have to communicate over PCIe peer-to-peer or even communicating via the CPU.

doctoboggan 4y ago

I know there seem to be a few different models to choose from, does anyone have a resource that collects them and shows the strengths and weaknesses of the various options? I’d be particularly interested in models optimized for apple silicon. Also, how do they compare to copilot?

Vetch 4y ago
Unfortunately, Copilot is a lot more capable. Most important is that it works with many more languages out of the box, is continuously updated, has more mathematical plus scientific knowledge and is better at understanding your comments. As for which to use, you'd want to take the largest model that can you can fine-tune (not necessarily on your machine) and fits comfortably in your machine. The small models aren't as flexible and are basically just a cleverer autocomplete. Copilot is a lot smarter than that and smarter than all models I've tested up to 6B. It can often cleverly work out a lot of local patterns for what you're doing and this is true surprisingly often for novel languages, math, understanding graph notation, categorization policies and much, much more. I'd gone in hoping I could remove my dependence on Copilot but left disappointed. The small models up to 6B just don't compare. I don't have the resources to run the 13B model which is probably behind anyways due to not having trained as long on as much as copilot (which is continuously updated). I also suspect Copilot has a great deal of hand-coded hacks and caching tricks to improve the user experience.
- moyix 4y ago
  Yep, I definitely agree that the 6B and below models are worse than Copilot. The 16B ones are pretty good! But IMO still undertrained compared to Copilot, and of course much less accessible (though see elsewhere in this comment section; I think INT8 and even INT4 should be doable – this won't help much with inference latency but it should help most people fit the 16B model locally). I have high hopes for the BigCode project too; they will get to take advantage of a lot of things that have been learned about how to train code models effectively. One last note – I think that many of the improvements to Copilot since its release are attributable to "prompt engineering" and client-side smarts about what context should be included; I'm not certain, but I can believe the underlying Codex model used hasn't changed much (if at all) since code-davinci-002.

dang 4y ago

Related: FauxPilot – an attempt to build a locally hosted version of GitHub Copilot - https://news.ycombinator.com/item?id=32327711 https://news.ycombinator.com/item?id=32327711 - Aug 2022 (79 comments)

daviding 4y ago

This seems to work ok on a 3080Ti, even under WSL2. Nice! For larger files I'm getting errors for maximum token + prompt length - is there an easy way to tweak these limits for the client? I think the Copilot client needed some overrides for Fauxpilot for this, so hoping Gitlab client has that too.

speedgoose 4y ago

Shouldn’t this be in a separate extension?

avip3d 4y ago
That would make sense... but them most people wouldn't install the base gitlab extension.
eddyg 4y ago
Came here to say this. I’d much rather see effort being put into other areas of the GitLab Workflow extension (like being able to approve / view approvals of MRs).

ddevault 4y ago

You'd think that GitLab would pause from chasing after GitHub's tail for at least long enough to question whether or not these features are ethical.