7 ms·
Show HN: Use Code Llama as Drop-In Replacement for Copilot Chat
Hi HN,
Code Llama was released, but we noticed a ton of questions in the main thread about how/where to use it — not just from an API or the terminal, but in your own codebase as a drop-in replacement for Copilot Chat. Without this, developers don't get much utility from the model.
This concern is also important because benchmarks like HumanEval don't perfectly reflect the quality of responses. There's likely to be a flurry of improvements to coding models in the coming months, and rather than relying on the benchmarks to evaluate them, the community will get better feedback from people actually using the models. This means real usage in real, everyday workflows.
We've worked to make this possible with Continue (https://github.com/continuedev/continue https://github.com/continuedev/continue) and want to hear what you find to be the real capabilities of Code Llama. Is it on-par with GPT-4, does it require fine-tuning, or does it excel at certain tasks?
If you’d like to try Code Llama with Continue, it only takes a few steps to set up (https://continue.dev/docs/walkthroughs/codellama https://continue.dev/docs/walkthroughs/codellama), either locally with Ollama, or through TogetherAI or Replicate's APIs.
- mchiang 3y agogood to see code llama supported through Continue already. Are you seeing good results with code llama yet?
- sestinj 3y agoIt's definitely a knowledgable model. I'm seeing a bit of trouble with stopping at the right point and managing chat context, but this is all a matter of prompt engineering. Very promising, and exciting work to be done!
- jstummbillig 3y ago> I'm seeing a bit of trouble with stopping at the right point and managing chat context, but this is all a matter of prompt engineering Can you explain what you mean by this, maybe provide some examples?
- sestinj 3y agoYeah, I asked it to write bubble sort in python, and it did so perfectly, but then decided to add a couple dozen parentheses to the end. I then followed up asking it to remove these, and it didn't seem to grasp that there was a conversation, as it just rewrote the same thing. All said, this was after like 5 minutes of playing with the 13B model, and was not using any kind of human/assistant formatting, hence the need for at least simple prompting to make that work (or fine-tuning if it isn't trained yet on conversational data)
- rtolsma 3y agoWhen will tools/models like these start integrating with code servers and linters in IDEs instead of just yielding supercharged autocomplete?
- baggiponte 3y agoThe guy TJ De Vries works at Sourcegraph + contributes to neovim and he’s building sg.nvim - a plug-in to hook Sourcegraph AND their code assistant Cody to neovim LSP.
- BaculumMeumEst 3y agosurely steve yegge will not let this go unanswered
- _steve_yegge_ 3y agoI guess I've been summoned. We're primarily focusing on VSCode, IntelliJ and Neovim for Cody. Of course I'll be working on an Emacs version, but that's kinda best-effort for now. As for the new crop of codegen models, they seem to be getting to parity with GPT/Claude/Bard-class models for code autocompletions, but not so much for other tasks. We're working on incorporating OSS models, but I'd be surprised if they're ready for prime-time this year. I think next year they'll be huge. Just my $0.02, take with a grain of salt. Shit moves fast.
- tjdevries 3y agoI think "best-effort" would be a good tagline for emacs
- BaculumMeumEst 3y agoexcuse me @dang a ban is in order
- BaculumMeumEst 3y ago
- Xiol32 3y agoWhy is it necessary to sign up for two external services if this is running the LLM locally?
- mchiang 3y agoNo need to sign up for any service to use Continue with Ollama. This will do the inference all locally. I believe if you want to do the inference using Replicate or Together, you’ll have to sign up for their services.
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- daemonologist 3y agoYou can cobble together an OpenAI-esque server locally with llama.cpp: https://github.com/ggerganov/llama.cpp/issues/2766 https://github.com/ggerganov/llama.cpp/issues/2766 and the script here: https://www.reddit.com/r/LocalLLaMA/comments/15ak5k4/short_guide_to_hosting_your_own_llamacpp_openai/ https://www.reddit.com/r/LocalLLaMA/comments/15ak5k4/short_g... and use Continue's GGML Model option to query it: https://continue.dev/docs/customization#local-models-with-ggml https://continue.dev/docs/customization#local-models-with-gg... (I've gotten this working, more or less, but don't have the hardware to make it practical so I can't give any feedback about Continue or the model.)
- smith7018 3y agoI wish someone would make an IntelliJ/Android Studio plugin for Code Llama (or another local llm). I know both platforms have their own AI features but that involves sending my code to their servers which I'm really not a fan of.
- javajosh 3y agoWell, you might hold your nose for a few days while Copilot helps you write the IntelliJ plugin for Code Llama. Now that would be a good hacker news post...
- sestinj 3y agoWe've done work to set Continue up to be a JetBrains plugin—it's a matter of just a) figuring out the basics of putting a webview in a JetBrains plugin and b) implementing this class (https://github.com/continuedev/continue/blob/main/extension/src/continueIdeClient.ts https://github.com/continuedev/continue/blob/main/extension/...) that communicates with the server. So long story short, we're planning on this soon and welcome PRs. It's a medium-sized task.
- freckletonj 3y agoUniteAI (https://github.com/freckletonj/uniteai https://github.com/freckletonj/uniteai) I think fits the bill for you. This is my project, where the goal is to Unite your AI-stack inside your editor (so, Speech-to-text, Local LLMs, Chat GPT, Retrieval Augmented Gen, etc). It's built atop a Language Server, so, while no one has made an IntelliJ client yet, it's simple to. I'll help you do it if you make a GH Issue!
- kateklink 3y agoat Refact we have Jetbrains plugin that works with a bunch of local code models https://github.com/smallcloudai/refact/ https://github.com/smallcloudai/refact/
- yangcheng 3y agobut for code llm to be useful , the local machine need to have some very powerful GPU
- say_it_as_it_is 3y agoThis would be great if I had a beefy machine with 32GB ram and 16 cores. The alternatives presented are to use either Together or Standard api's to run llama-code. Both have a free tier but eventually you wind up paying. Has anyone calculated the financials and determined that the cost of these alternatives is less than copilot? Cost aside, there's a concern of quality. I should just buy a super computer..
- ramesh31 3y ago>This would be great if I had a beefy machine with 32GB ram and 16 cores. Honestly this is a pretty standard workstation setup now. The days of 16GB being sufficient are pretty much over. RAM is cheap. And you can run the 7b model with well under that much.
- sestinj 3y agoIf you run Ollama locally, the cost is zero. With Replicate, you pay about $0.005 per second, so with an average of say 15 seconds per request to generate the full response, and 25 inputs per day, this comes out to ~$50/month. More expensive than classic Copilot, but also solving a different, potentially more valuable problem. Also likely that costs drop as we're seeing across the board. And at a minimum, the free tiers probably get you all the experimentation you need before figuring out which model is best :) (including Continue's free trial for GPT-4)
- agcat 3y agoCurious to learn about the different, potentially more valuable problem that you mentioned about? Is this for Replicate OR Continue?
- whimsicalism 3y agoYes, you likely will either have to pay a monthly fee or own compute yourself. Where did this expectation of free compute come from? Copilot costs too.
- Havoc 3y ago
- ramesh31 3y agoI just love this so much. It took what, maybe 6 hours for these model binaries to be publicly available? Never seen so much excitement in tech.
- dcre 3y agoI don't see how things like Cursor[0] (a VS Code fork with a nicer Codex UI) have a moat or chance of profitability in the face of things like this. If you can run models locally or your employer can spin up their own instance for internal use on a decent server, why is anybody paying OpenAI? And that leaves Cursor with no cash flow to skim off of. [0]: https://www.cursor.so/ https://www.cursor.so/
- blackkettle 3y agoAre you paying OpenAI? I’m not. Facebook’s midterm game on “massive” AI seems to be “commoditize the competition” and ATM it looks like a good bet.
- dcre 3y agoI am paying a couple bucks a month for GPT-4 API calls, but not for Copilot usage.
- zaptrem 3y agoTheir value proposition appears to be a fancy UI. Better/cheaper models would only improve their product and grow their addressable customer base.
- dcre 3y agoAnybody can copy that UI once they figure out what works. No moat.
- mlsu 3y agoI wouldn't discount good UI. I don't use Cursor, but consider that GPT-3 sat around for months and nobody was really talking about "AI doom" and "the lightcone" until OpenAI put a good chat UI layer over it. I would argue that actually capturing the value from these models lies mostly in the UI: allowing the user to seamlessly and quickly extract useful information from the model.
- kristianp 3y agoAre there any copilot-style plugins for Sublime Text? I still can't stomach the idea of vs code.
- gettodachoppa 3y agoWhy is this post flagged? I've yet to try code stuff with AI (even Copilot). How well do local models like Code Llama work compared to GPT? When I use ChatGPT, GPT 3.5 feels a decade behind GPT 4. Half the answers are flawed. It's only GPT4 that boosts my productivity to an impressive level. If you look at local text models for fiction or chat, they seem way behind even 3.5. See the examples table at https://docs.sillytavern.app/usage/faq/#what-do-you-mean-when-you-say-the-quality-depends-on-the-ai https://docs.sillytavern.app/usage/faq/#what-do-you-mean-whe... Code tasks are the sort of thing where I need as much accuracy as possible. Would something like Code Llama actually boost productivity or is this just "look, we can technically do it too, even if the result is awful" thing?