5 ms·
I'm not very plugged into how to use these models, but I do love and pay for both ChatGPT and GitHub Copilot. How does one take a model like this (or a smaller
by LVB 3y ago
I'm not very plugged into how to use these models, but I do love and pay for both ChatGPT and GitHub Copilot. How does one take a model like this (or a smaller version) and leverage it in VS Code? There's a dizzying array of GPT wrapper extensions for VS Code, many of which either seem like kind of junk (10 d/ls, no updates in a year), or just lead to another paid plan, at which point I might as well just keep my GH Copilot. Curious what others are doing here for Copilot-esque code completion without Copilot.
- petercooper 3y agohttps://continue.dev/ https://continue.dev/ is a good place to start.
- sestinj 3y agobeat me to the punch : )
- speedgoose 3y agoContinue doesn’t support tab completion like Copilot yet. A pull/merge request is being worked on: https://github.com/continuedev/continue/pull/758 https://github.com/continuedev/continue/pull/758
- sestinj 3y agoRelease coming later this week!
- just_testing 3y agoThanks! I'm going to recommend it for my students who are interested in tinkering with LLMs.
- sa-code 3y agoThank you for your efforts
- jondwillis 3y agoBonus points for being able to use local models!
- israrkhan 3y agoThis looks really good..
- dan_can_code 3y agoIt's great. It's super easy to install ollama locally, `ollama run <preferred model>`, change the continue config to point to it, and it just works. It even has an offline option by disabling telemetry.
- madeofpalk 3y ago> Windows coming soon ugh, not so easy.
- dan_can_code 3y agoYes, that is certainly a downside I forgot to mention. Sorry to get your hopes up.
- sestinj 3y agoI’ve been working on continue.dev, which is completely free to use with your own Ollama instance / TogetherAI key, or for a while with ours. Was testing with Codellama-70b this morning and it’s clearly a step up from other OS models
- dan_can_code 3y agoHow do you test a 70B model locally? I've tried to query, but the response is super slow.
- sestinj 3y agoPersonally I was testing with TogetherAI because I don't have the specs for a local 70b. Using quantized versions helps (Ollama's downloads 4-bit by default, you can get down to 2), but it would still require a higher-end Mac. Highly recommend Together, it runs quite quickly and is $0.9/million tokens
- d_sc 3y agoare there any docs on setting up togetherAI with continue.dev? would be interested in checking that out as an alternative to OpenAI for experimenting with larger models that won't run/run well on a m1 max.
- sestinj 3y agoDefinitely, here is a brief reference page for the Together provider: https://continue.dev/docs/reference/Model%20Providers/togetherllm https://continue.dev/docs/reference/Model%20Providers/togeth..., and a higher-level explanation of model configuration here: https://continue.dev/docs/model-setup/overview https://continue.dev/docs/model-setup/overview
- vwkd 3y agoWhat’s the advantage of Together? The price is about the price of GPT 3.5 Turbo ($1/mil tokens is $0.001/thousand tokens), which has the advantage of wide ecosystem and support.
- SparkyMcUnicorn 3y agoThere are some projects that let you run a self-hosted Copilot server, then you set a proxy for the official Copilot extension. https://github.com/fauxpilot/fauxpilot https://github.com/fauxpilot/fauxpilot https://github.com/danielgross/localpilot https://github.com/danielgross/localpilot
- israrkhan 3y agoI tried fauxpilot to make it work on my own llama.cpp instance, but didn't work out of the box. Filed a github issue, but did not get any traction. Eventually gave up on it. This was around 5 months ago. Things might have improved by now.
- water-data-dude 3y agoWhen I was setting up a local LLM to play with I stood up my own Open AI API compatible server using llama-cpp-python. I installed the Copilot extension and set OverrideProxyUrl in the advanced settings to point to my local server, but CoPilot obstinately refused to let me do anything until I’d signed in to GitHub to prove that I had a subscription. I don’t _believe_ that either of these lets you bypass that restriction (although I’d love to be proven wrong), so if you don’t want to sign up for a subscription you’ll need to use something like Continue.
- siilats 3y agoThis plugin just installs in Jetbrains IDE and sets up a local llama.cpp server https://plugins.jetbrains.com/plugin/21056-codegpt https://plugins.jetbrains.com/plugin/21056-codegpt
- marinhero 3y agoYou can download it and run it with [this](https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui). There's an API mode that you could leverage from your VS Code extension.
- cmgriffing 3y agoI've been using Cody by Sourcegraph and liking it so far. https://sourcegraph.com/cody https://sourcegraph.com/cody
- apapapa 3y agoFree Bard is better than free ChatGPT... Not sure about paid versions
- ignoramous 3y agoBard censorship is annoying. One thing I've found (free) Bard to better than the rest at is summarizing book chapters, manuals, and docs. It is also surprisingly good at translation (X to English), as it often adds context to what its translating. With careful prompt engineering, you can get a lot out of free Bard except when its censored.
- apapapa 3y agoNot sure why you talk about Bard's censorship and not Chatgpt's but cause in my experiences it is much worst.
- dxxvi 3y agoI'm learning Rust. It seems to me that Bard is better than OpenAI 4 I use for free at work.
- ado__dev 3y agoYou can try it with Sourcegraph Cody. https://sourcegraph.com/cody https://sourcegraph.com/cody And instructions on how to change the provider to use Ollama w/ whatever model you want: Install and run Ollama - Put ollama in your $PATH. E.g. ln -s ./ollama /usr/local/bin/ollama. - Download Code Llama 70b: ollama pull codellama:70b - Update Cody's VS Code settings to use the unstable-ollama autocomplete provider. - Confirm Cody uses Ollama by looking at the Cody output channel or the autocomplete trace view (in the command palette). - Update the cody settings to use "codellama:70b" as the ollama model https://github.com/sourcegraph/cody/pull/2635 https://github.com/sourcegraph/cody/pull/2635
- just_testing 3y agoHey, thanks for the tip. One issue, though: I took a look at the Cody website and it looks like one can't have unlimited completions even when self-hosting a LLM. I understand you guys have a business model and need to make money out of it. I'm just asking because I work as a teacher and I have students who can't pay an extra subscription and/or students who want to hack into stuff.
- raxxorraxor 3y agoI use the plugin Twinny in conjunction with ollama to host the models. Easy setup and quite powerful assistance. You need a decent rig though, since you don't want any latency for features like autocomplete. But even if you don't have a faster rig, you can still leverage it for slower tasks to generate docs or tests. Twinny should really be more popular, didn't find a more powerful no-bullshit plugin for VSCode.