9 ms·
LocalPilot: Open-source GitHub Copilot on your MacBook
- ShamelessC 3y agoQueue fifty comments about how language on the repository is misleading, “does this download Microsoft weights?” and other friends.
- rcarmo 3y agoWell, technically it is a MITM proxy for a locally hosted LLM, but people these days prefer simple, catchy names... But it might be useful if, say, you have a local GPU-powered machine on your LAN. I just wish they weren't using the advanced settings in the CoPilot extension and were using, say, one of the many OpenAI-powered alternatives (like Genie) -- would feel less like a hack and more like an alternative.
- notpushkin 3y agoYeah. My main problem with this is that the Copilot extension is proprietary (or at least it was – maybe that has changed?) and you can't install it on VSCodium, for example.
- rudasn 3y agoYup, still is.
- marci 3y agoAnd queue the somewhat relevant xkcd: https://xkcd.com/1053/ https://xkcd.com/1053/
- deleted 3y ago[deleted]
- imrehg 3y agoI guess similar to ollama (recently discussed: https://news.ycombinator.com/item?id=36802582 https://news.ycombinator.com/item?id=36802582) which also has support for code-focused models (see: https://ollama.ai/library https://ollama.ai/library). I tried pretty much all of them with Continue in VSCode, and it's a bit hit and miss, but the main difference is the way the workflows work (Copilot is mostly line completion, Continue is mostly chat or patches). So the main value add here for me would be a more Copilot-like workflow (which seems to align better with the day-to-day experience I has so far).
- fzzzy 3y agoThis is interesting because I started using continue specifically because it doesn't just do tab completion. I have normal Github Copilot turned on again as well now after having it off for a while and the tab completion does seem to be better than it was a few months ago. Still, explicitly asking continue to write a diff has a higher 'this is correct' rate for me because I can describe what I want in detail. Tab Completion seems fine for cases where I add one new line and the next two lines are just incredibly obvious. I am going to experiment with writing a little comment first to see if it primes the tab completion to do something non-obvious.
- imrehg 3y agoMaybe it depends on the work people are doing. I am doing a lot of data engineering where I can be often tabbing through while adjusting bits that are off, or doing Python portion by portion, and depending on the complexity it does go from "finish this string I'm writing" to few lines worth of code. I'm often surprised that how quickly the model discovers reasonable patterns (even across files as well, that's often necessary to be correct). With the diffs I find that by the time I describe things in a way it has a chance of working, I might have just written the whole thing myself. Especially as the diffs are often need correctly. With tabs that correction is part of a fast feedback look, with diffs it's so far a slower loop and just more awkward. Of course, with changes to workflows one or the other can shine and if there's an interface that's faster than typing the instructions out that might just supercharge things for the diff/chat type too.
- 3y ago
- amelius 3y agoWhy does it say "Use GitHub Copilot locally on your Macbook with one-click!" when it obviously doesn't use the Copilot model?
- Closi 3y agoBecause it is hijacking the Github Copilot functionality in VS Code. But yes, confusing notheless.
- BoorishBears 3y agoI'd really love someone to explain how it's confusing: As the parent says, it's dead obvious that it's not going to be GPT-4 running on your Macbook, the title starts by naming it something else, it aims for the very specific style of completion of Github Copilot... Normally these projects hijack the word 'local' by sending all of your data to a 3rd party API and that is confusing... but we finally get one that runs the model locally, does what it says on the tin, and some people still find a way to paint it as deception?
- Closi 3y agoIt’s confusing because it says “GitHub copilot running locally on your mac” when it is actually Llama running locally on your mac, but with the copilot interface. So the confusion is because what it says it is explicitly different to what it actually is, which understandably will be confusing.
- BoorishBears 3y agoBecause it obviously doesn't use the Copilot model: everyone will know what they mean.
- shim__ 3y agoDoes this require M2 AI capabilities or can it also run on other platforms?
- guywhocodes 3y agoSeems to be running on llama.cpp, so it's going to be a question of performance. I don't have any M-cpu but on my 13th gen i5 I can run mistral at about 6.5 tokens per second. Which seems comparable to what this is.
- SushiHippie 3y agoIt just uses llama.cpp as a "backend", so anywhere where llama.cpp works this should work too if I see this correctly.
- zerop 3y agoAnother OSS alternate - https://continue.dev/ https://continue.dev/
- shortrounddev2 3y agoWhy does it seem like a lot of the local AI tools target mac specifically? Why don't AI developers seem to be able to write cross platform software?
- simcop2387 3y agoIt's because of a quirk of the hardware. The Unified Memory setup that Apple built into the M2 (and M1? not sure) systems means that you have high bandwidth and large amounts of memory available to the GPU inside them, which lets you run LLMs very efficiently, and very easily. Combined with the fact that that part of the hardware is the same across all the machines it makes for a very easy target to get a lot built with minimal effort.
- blitzar 3y agoQuirk / Deliberate design decision Potato / potato
- deleted 3y ago[deleted]
- paulmd 3y agoThe matrix multiplication hardware and data types aren’t standardized yet. Also, the M1 Max has more bandwidth than an epyc Milan to actually feed all of that. It’s about the same bandwidth as a PS5, but in a mobile package, with none of the latency of GDDR6. Much more powerful than a standard dual-channel consumer cpu.
- Havoc 3y agoHaven’t managed to get the official copilot extension to use local anything. Always seems to ask for a login.
- shortrounddev2 3y agoYou have to pay monthly for it
- Havoc 3y agoIf I'm paying for the service then I wouldn't bother using the inferior local hosted one...
- wokwokwok 3y agoHm… the q4 34B code llama (which is used here) performs quite poorly in my experience. Using a high quantised larger model gives you an unrealistic impression that smaller models and larger models are roughly equivalently capable… but it’s a trade off. The larger codellama model is categorically better, if you don’t lobotomise it. It’d be better if instead of making opinionated choices (which aren’t great) it guided you on how to select an appropriate model…
- yustee 3y agoUnlike 7B & 13B, the 34B model does not have code infilling which is very useful for code completion
- raincole 3y agoI found the fact Copilot is close sourced -- not just the model, but even the plugins are close sourced -- is very worrying. Good to see efforts on the alternatives.
- 3abiton 3y agoThere is also Phind and WizardLM, but I found their performance to be quite poor tbh. It's hard to match gpt-3.5 quality.
- lolinder 3y agoCan you elaborate on why it's worrying to you? Is it more worrying than, say, Visual Studio being closed source?
- manojlds 3y agoYes because there's no reason for a visual studio code extension like this to be closed source.
- raincole 3y agoIt's worrying because it feels like MS will add more and more closed source "features" to VSCode and undermine its open source-ness. It also prevent other editors' users from building Copilot plugins. For example, there won't be a Copilot plugin for Emacs that can be accepted by Emacs's official repostiory. > Visual Studio being closed source If VScode isn't the de facto universal editor accepted by every programming language's community (notice that even this particular thread is about a VSCode plugin!), I won't be so worried.
- adocomplete 3y agoAnd if you're looking for alternatives - we at Sourcegraph are building Cody - and it is open source. https://about.sourcegraph.com/cody https://about.sourcegraph.com/cody
- all2 3y agoI tried to use Cody and the UI was confusing and the completions were unbearably slow. I was using the Rider plugin. I want Cody to work for me, but right now it doesn't. I really, really want whole project awareness (maybe even to leverage a concurrently running language server?) for my completions. Do you guys have a usage tutorial or a video somewhere? Are you flexible with how your UI is being implemented (ie, can I pitch you ideas)?
- SushiHippie 3y agoFWIW: you can use any other proxy server for this to any openai compatible api server. e.g. with mitmproxy and llama-cpp-python server python -m llama_cpp.server --n_ctx 4096 --n_gpu_layers 1 --model ./path/to/..gguf and then with mitmproxy in another terminal mitmproxy -p 5001 --mode reverse:http://127.0.0.1:8000 and then set this in your vscode settings.json (the same as for localpilot): "github.copilot.advanced": { "debug.testOverrideProxyUrl": "http://localhost:5001", "debug.overrideProxyUrl": "http://localhost:5001" } works way better for me than localpilot
- cgrs 3y agoJust out of curiosity, why do you need a reverse proxy? Couldn't you connect to that openai-compatible server directly?
- SushiHippie 3y agoThanks, I don't know why I didn't think of that. Just tried it and it works at least with the llama-cpp-python server, but I suppose some other servers and very likely hosted services (SNI, certificate subjects, ...) would have problems with that.
- newman314 3y agoI'm assuming that this means that I could also potentially connect to a local more powerful box?
- SushiHippie 3y agoYup
- hoschicz 3y agoHow is context-from-following-tokens implemented in the pure OAI API? I assumed there must be specialized models that have two separate context windows.
- james2doyle 3y agoLooks cool! Always like to see these local alternatives. I'm a Sublime Text user (it is still amazing!) so there aren't many options for LLM assistants. The only one I found that works for me on Sublime is https://codeium.com/ https://codeium.com/ and it is also free for the basic usage. They have a great list of supported editors: - Android Studio - Chrome (Colab, Jupyter, Databricks and Deepnote, JSFiddle, Codepen, Codeshare, and StackBlitz) - CLion - Databricks - Deepnote - Eclipse - Emacs - GoLand - Google Colab - IntelliJ - JetBrains - Jupyter Notebook - Neovim - PhpStorm - PyCharm - Sublime Text - Vim - Visual Studio - Visual Studio Code - WebStorm - Xcode I have found that the completions are decent enough. I do find that sometimes the completion suggestions are too aggressive and try to complete more than I want so I end up leaving it off until I feel like I could use it.
- yaroslavyar 3y agoHey, check the one that I made[1]. It isn’t the code completion assistant like the one you mentioned above, and it probably never will be. I see it more as a perfect coding companion, that is always under your fingertips and relieves you of googling most of the times. Yet it’s tied with OpenAI, and you have to pay it by yourself, but the former should be changed rather sooner than later. Bonus: in develop branch there is some-kind-of release candidate that a way more robust that the current release is. [1]: https://github.com/yaroslavyaroslav/OpenAI-sublime-text https://github.com/yaroslavyaroslav/OpenAI-sublime-text
- mcbuilder 3y agoOkay, I actually got local co-pilot set up. You will need these 4 things. 1) CodeLlama 13B or another FIM model https://huggingface.co/codellama/CodeLlama-13b-hf https://huggingface.co/codellama/CodeLlama-13b-hf. You want "Fill in Middle" models because you're looking at context on both sides of your cursor. 2) HuggingFace llm-ls https://github.com/huggingface/llm-ls https://github.com/huggingface/llm-ls A large language mode Language Server (is this making sense yet) 3) HuggingFace inference framework. https://github.com/huggingface/text-generation-inference https://github.com/huggingface/text-generation-inference At least when I tested you couldn't use something like llama.cpp or exllama with the llm-ls, so you need to break out the heavy duty badboy HuggingFace inference server. Just config and run. Now config and run llm-ls. 4) Okay, I mean you need an editor. I just tried nvim, and this was a few weeks ago, so there may be better support. My expereicen was that is was full honest to god copilot. The CodeLlama models are known to be quite good for its size. The FIM part is great. Boilerplace works so much easier with the surrounding context. I'd like to see more models released that can work this way.
- Tokumei-no-hito 3y agowhat hardware is needed to run this? could a MBP handle it?
- thebiss 3y agoPer the repo, it runs certain models on a Macbook Pro with an Apple M2 Max.
- BaculumMeumEst 3y agoI would love to be able to take a base model and fine-tune it on a handful of hand picked repositories that are A) in a specific language I want to use and B) stylistically similar to how I want to write code. I’m not sure how possible that is to do, but I hope we can get there at some point.