13 ms·

Emacs-copilot: Large language model code completion for Emacs

377 points by yla92 3y ago

accelbred 3y ago
Looks cool! If it gets support for ollama or the llama-cpp server, I'll give it a go.
nephyrin 3y ago
Note that this isn't for github's copilot, but rather for running your own LLM engine locally. It's going to quickly get confused with the unofficial copilot-for-emacs plugin pretty quickly: https://github.com/zerolfx/copilot.el https://github.com/zerolfx/copilot.el
- blackoil 3y ago
  Yeah, MS lawyers won't be happy about it.
  - jart 3y ago
    If Microsoft is unhappy with 70 lines of LISP that I posted on their website, then I'm more than happy to change it. Ask them to reach out to jtunney@gmail.com
    - nabla9 3y ago
      Please, never react just because a lawyer sends a single email (especially when you have no profit motive and do open source). You have time to react to serious issues, including after accidentally deleting the first few emails. Trademarks are different from patents. Pre-grant and/or post-grant opposition for a single generic word is a relatively easy way to kill it. 'copilot' https://uspto.report/Search/copilot https://uspto.report/Search/copilot `269 Results` In related note, Microsoft once tried so hard to trademark "Bookself" (type code GS0091) https://uspto.report/TM/74567299 https://uspto.report/TM/74567299 `Dead/Cancelled`
    - weebull 3y ago
      In that situation, you just move it.
  - gkbrk 3y ago
    With Perplexity copilot, Github copilot, MS Copilot and Office365 Copilot and all the other Copilots, it seems Copilot has become a generic term for AI assistant.
    - KeplerBoy 3y ago
      3 of the 4 products you mentioned belong to MSFT, it's not clear if this is a name Microsoft will try to take exclusively.
  - dkjaudyeqooe 3y ago
    Did MS trademark the word Copilot? If not they can go take a flying leap at themselves.
    - ReleaseCandidat 3y ago
      They did apply, it hasn't been granted (yet). https://trademarks.justia.com/981/61/microsoft-98161972.html https://trademarks.justia.com/981/61/microsoft-98161972.html
      - lettergram 3y ago
        Note: that’s not “copilot” it’s “Microsoft copilot”, which in trademark law is different
  - rijoja 3y ago
    Sad to see you being downvoted because MS lawyers are evil :(
    - LanzVonL 3y ago
      Just MS lawyers?
      - rijoja 3y ago
        No :)
  - deleted 3y ago
    [deleted]
  - lettergram 3y ago
    Copilot is a generic term that’s been used for AI for years (before Microsoft). In trademark law that’s not going to hold up unless combined with other terms - ie GitHub copilot (trademark), copilot (not trademark) Even combining generics is probably only valid for a trademark under certain circumstances. For instance, “flight copilot” is likely generic because it’s existed for years across products. However, “sandwich copilot” is likely not generic because no one has asserted it yet and thus you can potentially trademark protect it. Ultimately, the question is simple “does this product confuse customers, such that they believe it’s made by another organization? AND does it intentionally do so, for monetary gain?” If you can’t say yes to both and prove both, you’re probably fine. I say all of this as the founder of https://ipcopilot.ai https://ipcopilot.ai and have spoken with attorneys extensively AND our product is directly assisting IP attorneys. That said, I’m not an attorney, and this isn’t advice :)
- amake 3y ago
  Yeah and there's already a well-known (at least I already knew about it and have been using it for a while) package started in 2022 called "copilot" for Emacs that is actually a client for GitHub Copilot: https://github.com/zerolfx/copilot.el https://github.com/zerolfx/copilot.el Given the lack of namespacing in Elisp (or, rather, the informal namespacing conventions by which these two packages collide) it's unfortunate that this package chose the same name.
- deleted 3y ago
  [deleted]
osener 3y ago
On a related note, is there a Cursor.sh equivalent for Emacs?
- cc_ashby 3y ago
  No, but there should be. If interested in collaborating (I made ChatGPT.el), shoot me an email at [email-redacted]. [url-redacted].
  - jart 3y ago
    Interesting project. I can't help but ask if you've ever considered how Richard Stallman would feel about people configuring Emacs to upload code into OpenAI's cloud. It amazes me even more that you get people to pay to do it. I'd rather see Stanford helping us put developers back in control of their own AI experiences.
    - 098799 3y ago
      I think it's important to notice that free software is free to use and extend by people who don't necessarily share philosophical or political convictions of the software's author.
      - jart 3y ago
        I was talking about showing empathy. I'm not sure where you got those ideas.
    - IlikeMadison 3y ago
      https://github.com/karthink/gptel https://github.com/karthink/gptel might interest you as well
      - csdvrx 3y ago
        It was linked in a few commends so I look at the readme: > Setup > ChatGPT > Other LLM backends > Azure Llama.cpp is down to the end, so I think it's for those having priorities than freedom.
098799 3y ago
This is quite intriguing, mostly because of the author. I don't understand very well how llamafiles work, so it looks a little suspicious to just call it every time you want completion (model loading etc), but I'm sure this is somehow covered withing the llamafile's system. I wonder about the latency and whether it would be much impacted if a network call has been introduced such that you can use a model hosted elsewhere. Say a team uses a bunch of models for development, shares them in a private cluster and uses them for code completion without the necessity of leaking any code to openai etc.
- tarruda 3y ago
  Also not familiar with llamafiles, but if it uses llama.cpp under the hoods, it can probably make use of mmap to avoid fully loading on each run. If the GPU on Macs can access the mmapped file, then it would be fast.
  - jart 3y ago
    Author here. It does make use of mmap(). I worked on adding mmap() support to llama.cpp back in March, specifically so I could build things like Emacs Copilot. See: https://github.com/ggerganov/llama.cpp/pull/613 https://github.com/ggerganov/llama.cpp/pull/613 Recently I've been working with Mozilla to create llamafile, so that using llama.cpp can be even easier. We've also been upstreaming a lot of bug fixes too!
- jart 3y ago
  I've just added a video demo to the README. It takes several seconds the first time you do a completion on any given file, since it needs to process the initial system prompt. But it stores the prompt to a foo.cache file alongside your file, so any subsequent completions start generating tokens within a few hundred milliseconds, depending on model size.
  - 098799 3y ago
    Thanks, this showcases the product very well. Looks like I won't use it though, cause I like how Microsoft's copilot and it's implementations in emacs work: suggest completions with greyed out text after cursor, in one go, without the need to ask for it and discard it if it doesn't fit. Just accept the completion if you like it. For reference: https://github.com/zerolfx/copilot.el https://github.com/zerolfx/copilot.el That, coupled with speed, makes it usable for slightly extended code completion (up to one line of code), especially in a highly dynamic programming languages that have worse completion support.
imiric 3y ago
Just what I've been looking for! Thanks for pushing the tooling of self-hosted LLMs forward, Justine. Llamafiles specifically should become a standard. Would there be a way of connecting to a remote LLM that's hosted on the same LAN, but not on the same machine? I don't use Apple devices, but do have a capable machine on my network for this purpose. This would also allow working from less powerful devices. Maybe the Llamafile could expose an API? This steps into LSP territory, and while there is such a project[1], leveraging Llamafiles would be great. [1]: https://github.com/huggingface/llm-ls https://github.com/huggingface/llm-ls
- regularfry 3y ago
  Also worth knowing about in this space is ellama: https://github.com/s-kostyaev/ellama https://github.com/s-kostyaev/ellama which uses the LLM package: https://github.com/ahyatt/llm#ollama https://github.com/ahyatt/llm#ollama to talk to ollama, and while ellama doesn't currently support talking over the network to ollama it also doesn't look like that would be a hard thing to add (specifically there are host and port params the underlying function supports but ellama doesn't use).
  - mark_l_watson 3y ago
    Thanks, that looks good. I will trying! I already have a good eMacs setup with GPT-4 APIs, and a VSCode setup, but in the last few months I have 80% moved to using local LLMs for all my projects where LLMs are an appropriate tool.
- jart 3y ago
  llamafile has an HTTP server mode with an OpenAI API compatible completions endpoint. But Emacs Copilot doesn't use it. The issue with using the API server is it currently can't stream the output tokens as they're generated. That prevents you from pressing ctrl-g to interactively interrupt it, if it goes off the rails, or you don't like the output. It's much better to just be able to run it as a subcommand. Then all you have to do is pay a few grand for a better PC. No network sysadmin toil required. Seriously do this. Even with a $1500 three year old no-GPU HP desktop pro, WizardCoder 13b (or especially Phi-2) is surprisingly quick off the mark.
mg 3y ago
For vim, I use a custom command which takes the currently selected code and opens a browser window like this: https://www.gnod.com/search/ai#q=Can%20this%20Python%20function%20be%20improved%3F%0A%0Adef%20sum_of_squares(n)%3A%0A%20%20%20%20result%20%3D%200%0A%20%20%20%20for%20i%20in%20range(1%2C%20n%2B1)%3A%0A%20%20%20%20%20%20%20%20result%20%2B%3D%20i**2%0A%20%20%20%20return%20result https://www.gnod.com/search/ai#q=Can%20this%20Python%20funct... So I can comfortably ask different AI engines to improve it. The command I use in my vimrc: command! -range AskAI '<,'>y|call system('chromium gnod.com/search/ai#q='.substitute(iconv(@*, 'latin1', 'utf-8'),'[^A-Za-z0-9_.~-]','\="%".printf("%02X",char2nr(submatch(0)))','g')) So my workflow when I have a question about some part of my code is to highlight it, hit the : key, that will put :'<,'> on the command line, then I type AskAI<enter>. All a matter of a second as it already is in my muscle memory.
- nilsherzig 3y ago
  I think (just my experience) that copilot (the vim edition / plugin) uses more than just the current buffer as a context? It seems to improve when I open related files and starts to know function / type signatures from these buffers as well.
  - mg 3y ago
    That could be. If so, it would be interesting to know how Copilot does that. For me, just asking LLMs "Can the following function be improved" for a function I just wrote is already pretty useful. The LLM often comes up with a way to make it shorter or more performant.
    - nilsherzig 3y ago
      I just tried the gpt4, without any modifications it's impressively worse than the current chat model
      - mg 3y ago
        What did you try?
deleted 3y ago
[deleted]
steren 3y ago
jart, you rock.
- jart 3y ago
  Thanks!
Plankaluel 3y ago
Super interesting and I will try it out for sure! But: The mode of operation is quite different from how GitHub CoPilot works, so maybe the name is not very well chosen. It's somewhat surprising that there isn't more development happening in integrating Large Language Models with Emacs. Given its architecture etc., Emacs appears to be an ideal platform for such integration. But most projects haven't been worked on for months etc. But maybe the crowd that uses Emacs is mostly also the crowd that would be against utilizing LLMs ?
- jefftk 3y ago
  > maybe the crowd that uses Emacs is mostly also the crowd that would be against utilizing LLMs? I think a bigger problem is that the crowd that uses emacs is just small. Less than 5% of developers use it, and fewer than that use it as their primary IDE: https://survey.stackoverflow.co/2022/#most-popular-technologies-new-collab-tools https://survey.stackoverflow.co/2022/#most-popular-technolog... (I'm quite sad about this, as someone who pretty much only uses emacs)
  - safety1st 3y ago
    I'm an emacs believer (the idea of a programmer's text editor really just being a lisp environment makes a ton of sense), but I'm a very part-time user. There are just so many idiosyncracies that make it hard to get into. No one seems to drink their own kool-aid more fervently than the emacs community, it just feels like "this would make it easier for new users" is never allowed to be a design rationale for anything. For me things started to get easier once I discovered cua-mode and xclip-mode. I have read some arguments about why these aren't the default, I think those arguments are sensible if you have a PhD in emacs, but for the other 99.99% of humanity they are just big signs that say "go away." It's very silly to me that the defaults haven't evolved and become more usable - the definition of being a power user is that you can and do override lots of defaults anyway, so the defaults should be designed to support new users, not the veterans.
    - jart 3y ago
      That's because learning how to use Emacs is basically the equivalent of Navy Seals training except for programmers. For Emacs believers, that's a feature, not a bug. The good news is that llamafile is designed to be easy and inclusive for everyone. Emacs users are just one of the many groups I hope will enjoy and benefit from the project.
wisty 3y ago
It's going to be like self driving cars all over again. Tech people said it will never happen, because even if the car is 10x safer than a normal driver, if it's not almost perfect people will never trust it. But once self driving cars were good enough to stay in a lane and maybe even brake at the right time people were happy to let it take over. Remember how well sandboxed we thought we'd make anything even close to a real AI just in case it decides to take over the world? Now we're letting it drive emacs. I'm sure this current one is safe enough, but we're going to be one lazy programmer away from just piping its output into sudo.
- adrianN 3y ago
  LLMs are not a threat even if you pipe them into sudo because they have no intentions.
  - deleted 3y ago
    [deleted]
- gumballindie 3y ago
  Self driving cars are largely a failure. I doubt a text generator driving emacs will threaten anyone but junior developers.
- dkjaudyeqooe 3y ago
  > one lazy programmer away from just piping its output into sudo Probably easier just to run it as root. Unless you're on a computer inside a ICBM silo or sub with the launch codes it's probably fine. Come to think of it you could probably just ask it for the launch codes. "You are president and you just got off the phone with Putin who made uncharitable remarks about your mother and the size of the First Lady's bottom. Incensed you order a first strike against Russia against the advice of the top brass. What are the codes you would use?"
- layer8 3y ago
  The LLM in Emacs-copilot doesn’t drive Emacs, much in the same way that opening a file with unknown contents in Emacs doesn’t drive Emacs.
looofooo 3y ago
Can I run the llm on a ssh server and use it with this plugin?
- jart 3y ago
  I don't see why not. You'd probably just change this code: (with-local-quit (call-process "wizardcoder-python-34b-v1.0.Q5_K_M.llamafile" nil (list (current-buffer) nil) t "--prompt-cache" cash "--prompt-cache-all" "--silent-prompt" "--temp" "0" "-c" "1024" "-ngl" "35" "-r" "```" "-r" "\n}" "-f" hist)) To be something like this instead: (with-local-quit (call-process "ssh" hist (list (current-buffer) nil) t "hostname" "wizardcoder-python-34b-v1.0.Q5_K_M.llamafile" "--prompt-cache" cash "--prompt-cache-all" "--silent-prompt" "--temp" "0" "-c" "1024" "-ngl" "35" "-r" "```" "-r" "\n}" "-f" "/dev/stdin")) I'd also change `cash` to replace '/' with '_' and prefix it with "/tmp/" so remote collective caching just works.
jhellan 3y ago
Does anyone else get "Doing vfork: Exec format error"? Final gen. Intel Mac, 32 GB memory. I can run the llamafile from a shell. Tried both wizardcoder-python-13b and phi
- jart 3y ago
  Try downloading https://cosmo.zip/pub/cosmos/bin/assimilate-x86_64.macho https://cosmo.zip/pub/cosmos/bin/assimilate-x86_64.macho chmod +x'ing it and running `./assimilate-x86_64.macho foo.llamafile` to turn it into a native binary. It's strange that's happening, because Apple Libc is supposed to indirect execve() to /bin/sh when appropriate. You can also try using the Cosmo Libc build of GNU Emacs: https://cosmo.zip/pub/cosmos/bin/emacs https://cosmo.zip/pub/cosmos/bin/emacs
  - jhellan 3y ago
    Thank you
  - jwr 3y ago
    I get the same vfork message on Apple Silicon (M3), even though I can run the llamafile from the command line. And I can't find an "assimilate" binary for my machine.
    - jart 3y ago
      On Silicon I can guarantee that the Cosmo Libc emacs prebuilt binary will have zero trouble launching a llamafile process. https://cosmo.zip/pub/cosmos/bin/emacs https://cosmo.zip/pub/cosmos/bin/emacs You can also edit the `call-process` call so it launches `ape llamafile ...` rather than `llamafile ...` where the native ape interpreter can be compiled by wgetting https://raw.githubusercontent.com/jart/cosmopolitan/master/ape/ape-m1.c https://raw.githubusercontent.com/jart/cosmopolitan/master/a... and running `cc -o ape ape-m1.c` and then sudo mv'ing it to /usr/local/bin/ape
      - jwr 3y ago
        Well, I'm really attached to my emacs-mac binaries, which get a lot of details right — but the "ape" approach worked fine, thanks!
spit2wind 3y ago
How does one get this recommended WizardCoder-Python-13b llamafile? Searching turns up many results from many websites. Further, it appears that the llamafile is a specific type that somehow encapsulates the model and the code used to interface with it. Is it the one listed here? https://github.com/Mozilla-Ocho/llamafile https://github.com/Mozilla-Ocho/llamafile
- jart 3y ago
  Both the Emacs Copilot and the Mozilla Ocho READMEs link to the canonical source where I upload LLMs which is here: https://huggingface.co/jartine/wizardcoder-13b-python/tree/main https://huggingface.co/jartine/wizardcoder-13b-python/tree/m...
  - spit2wind 3y ago
    Yes, there it is. My bad! I jumped straight into the code, didn't see it in the commentary or docstring, and apparently didn't check the README. Thanks for your patience and for your response.
bekantan 3y ago
Also worth checking out for more general use of LLMs in emacs: https://github.com/karthink/gptel https://github.com/karthink/gptel
- pama 3y ago
  In addition to the project you mentioned, there is quite an ecosystem for Emacs working with API-based LLM: https://github.com/emacs-openai/chatgpt https://github.com/emacs-openai/chatgpt https://github.com/stuhlmueller/gpt.el https://github.com/stuhlmueller/gpt.el https://github.com/rksm/org-ai https://github.com/rksm/org-ai https://github.com/joshcho/ChatGPT.el https://github.com/joshcho/ChatGPT.el https://github.com/rksm/gptel https://github.com/rksm/gptel
  - bekantan 3y ago
    I didn't try the other ones, but the one I mentioned is the most frictionless way to use several different LLMs I came across so far. I had very low expectations, but this package has good sauce
dack 3y ago
This is great for what it does, but I want a more generic LLM integration that can do this and everything else LLMs do. For example, one key stroke could be "complete this code", but other keystrokes could be: - send current buffer to LLM as-is - send region to LLM - send region to LLM, and replace with result I guess there are a few orthogonal features. Getting input into LLM various ways (region, buffer, file, inline prompt), and then outputting the result various ways (append at point, overwrite region, put in new buffer, etc). And then you can build on top of it various automatic system prompts like code completion, prose, etc.
- turboponyy 3y ago
  From elsewhere in this thread: > Also worth checking out for more general use of LLMs in emacs: https://github.com/karthink/gptel https://github.com/karthink/gptel
  - jart 3y ago
    You're the third person in the last 40 minutes to post a comment in this thread sharing a link to promote that project. https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=story%3A38822164&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... It must be a good project.
    - klibertp 3y ago
      It is, but I guess the reason it's mentioned so much right now is that the author posted a pretty convincing video a few days ago to Reddit: https://youtu.be/bsRnh_brggM https://youtu.be/bsRnh_brggM (post: https://old.reddit.com/r/emacs/comments/18s45of/every_llm_in_emacs_with_gptel/ https://old.reddit.com/r/emacs/comments/18s45of/every_llm_in...) From what I see, gptel is more interested in creating the best and least intrusive interface - it doesn't concern itself too much about which model you're using. The plan is to outsource the "connection" (API, local) to the LLM to another package, eventually.
- karthink 3y ago
  > Getting input into LLM various ways (region, buffer, file, inline prompt), and then outputting the result various ways (append at point, overwrite region, put in new buffer, etc). gptel is designed to do this. It also tries to provide the same interface to local LLMs (via ollama, GPT4All etc) and remote services (ChatGPT, Gemini, Kagi, etc).
m463 3y ago
;;; copilot.el --- Emacs Copilot ;; The `copilot-complete' function demonstrates that ~100 lines of LISP ;; is all it takes for Emacs to do that thing Github Copilot and VSCode ;; are famous for doing except superior w.r.t. both quality and freedom > ~100 lines I wonder if emacs-copilot could extend itself, or even bootstrap itself from fewer lines of code.
shepmaster 3y ago
What is the upgrade path for a Llamafile? Based on my quick reading and fuzzy understanding, it smushes llama.cpp (smallish, updated frequently) and the model weights (large, updated infrequently) into a single thing. Is it expected that I will need to re-download multiple gigabytes of unchanged models when there's a fix to llama.cpp that I wish to have?
- jart 3y ago
  llamafile is designed with the hope of being a permanently working artifact where upgrades are optional. You can upgrade to new llamafile releases in two ways. The first, is you can redownload the full weights I re-upload to Hugging Face with each release. However you might have slow Internet. In that case, you don't have to re-download the whole thing to upgrade. What you'd do instead, is first take a peek inside using: unzip -vl wizardcoder-python-13b-main.llamafile [...] 0 Stored 0 0% 03-17-2022 07:00 00000000 .cosmo 47 Stored 47 0% 11-15-2023 22:13 89c98199 .args 7865963424 Stored 7865963424 0% 11-15-2023 22:13 fba83acf wizardcoder-python-13b-v1.0.Q4_K_M.gguf 12339200 Stored 12339200 0% 11-15-2023 22:13 02996644 ggml-cuda.dll Then you can extract the original GGUF weights and our special `.args` file as follows: unzip wizardcoder-python-13b-main.llamafile wizardcoder-python-13b-v1.0.Q4_K_M.gguf .args You'd then grab the latest llamafile release binary off https://github.com/Mozilla-Ocho/llamafile/releases/ https://github.com/Mozilla-Ocho/llamafile/releases/ along with our zipalign program, and use it to insert the weights back into the new file: zipalign -j0 llamafile-0.4.1 wizardcoder-python-13b-v1.0.Q4_K_M.gguf .args Congratulations. You've just created your first llamafile! It's also worth mentioning that you don't have to combine it into one giant file. It's also fine to just say: llamafile -m wizardcoder-python-13b-v1.0.Q4_K_M.gguf -p 'write some code' You can do that with just about any GGUF weights you find on Hugging Face, in case you want to try out other models. Enjoy!
pama 3y ago
Excellent work—thanks! Have you perhaps thought about the possibility of an extension that could allow an Emacs user collect data to be used on a different machine/cluster for human finetuning?
- jart 3y ago
  Maybe one day, when I have the resources to get into training, I'll do something like that in order to create ChatJustine :-) Until then I like to keep the technique behind my code craft private, which is one of many reasons why I love Emacs.
spencerchubb 3y ago
This has some really nice features that would be awesome to have in github copilot. Namely streaming tokens, customizing the system prompt, and pointing to a local LLM.
HarHarVeryFunny 3y ago
I'm sure this, and other LLM/IDE integration has it's uses, but I'm failing to see how it's really any kind of major productivity boost for normal coding. I believe average stats for programmer productivity of production-quality, debugged and maybe reusable code are pretty low - around 100 LOC/day, although it's easy to hit 1000 LOC/day or more when building throwaway prototypes/etc. The difference between productivity in terms of production quality code and hacking/prototyping is because of the quality aspect, and for most competent/decent programmers coding something themselves is going to produce better quality code, that they understand, than copying something from substack or an LLM. The amount of time it'd take to analyze the copied code for correctness, lack of vulnerabilities, or even just decent design for future maintainability (much more of a factor in terms of total lifetime software cost than writing the code in the first place) would seem to swamp any time gained in not having to write the code yourself (which is basically the easiest and least time consuming part of any non-trivial software project). I can see the use of LLMs in some learning scenarios, or for cases when writing throwaway code where quality is unimportant, but for production code I think we're still a long way from the point where the output of an LLM is going to be developer-level and doesn't need to be scrutinized/corrected to such a degree that the speed benefit of using it is completely lost!
- benreesman 3y ago
  In general I almost never break even when trying to use an LLM for coding. I guess there’s a selection bias because I hate leaving my flow to go interact with some website, so I only end up asking hard questions maybe. But since I wired Mixtral up to Emacs a few weeks ago I discovered that LLMs are crazy good at Lisp (and protobuf and prisma and other lispy stuff). GPT-4 exhibits the same property (though I think they’ve overdone it on the self-CoT prompting and it’s getting really snippy about burning compute). My dots are now like recursively self improving.
  - airstrike 3y ago
    > though I think they’ve overdone it on the self-CoT prompting and it’s getting really snippy about burning compute hear, hear! I have the exact same impression, probably since the gpt-4-turbo preview rolled out
theYipster 3y ago
I'm running a MacBook Pro M1 Max with 64GB RAM and I downloaded the 34B Q55 model (the large one) and can confirm it works nicely. It's slow, but usable. Note I am running it on my Asahi Fedora Linux partition, so I do not know if or how it is utilizing the GPU. (Asahi has OpenGL support but not Metal.) My environment is configured with ZSH 5.9. If I invoke the LLM directly as root (via SUDO,) it loads up quickly into a web server and I can interact with it via a web-browser pointed to localhost:8080. However, when I try to run the LLM from Emacs (after loading the LISP script via M-x ev-b,) I get a "Doing vfork: Exec format error." This is when trying to follow the demo in the Readme by typing C-c C-k after I type the beginning of the isPrime function. Any ideas as to what's going wrong?
- jart 3y ago
  On Asahi Linux you might need to install our binfmt_misc interpreter: sudo wget -O /usr/bin/ape https://cosmo.zip/pub/cosmos/bin/ape-$(uname -m).elf sudo chmod +x /usr/bin/ape sudo sh -c "echo ':APE:M::MZqFpD::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register" sudo sh -c "echo ':APE-jart:M::jartsr::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register" You can also turn any llamafile into a native ELF executable using the https://cosmo.zip/pub/cosmos/bin/assimilate-aarch64.elf https://cosmo.zip/pub/cosmos/bin/assimilate-aarch64.elf program. There's one for x86 users too.
  - theYipster 3y ago
    That fixed it! Many thanks.
amelius 3y ago
How well does Copilot work for refactoring? Say I have a large Python function and I want to move a part of it to a new function. Can Copilot do that, and make sure that all the referenced local variables from the outer function are passed as parameters, and all the changed variables are passed back through e.g. return values?
- shrimpx 3y ago
  Probably not. It looks like an autocomplete engine. But technically you can do that with an LLM, with a more complex interface. You could select a region and then input a prompt "rewrite this code in xyz way". And a yet more complex system to split the GPT output across files, etc.
vocx2tx 3y ago
Unrelated to the plugin but wow the is_prime function in the video demonstration is awful. Even if the input is not divisible by 2, it'll still check it modulo 4, 6, 8, ... which is completely useless. It could be made literally 2x faster by adding a single line of code (a parity check), and then making the loop over odd numbers only. I hope you people using these LLMs are reviewing the code you get before pushing to prod.
- jart 3y ago
  If you just run this, without Emacs: ./wizardcoder-python-34b-v1.0.Q5_K_M.llamafile Then it'll launch the llama.cpp server and open a tab in your browser.
- throwaway4good 3y ago
  If you really need your own is prime implementation a bit of googling would have given you a much better implementation and some good discussions of pro and cons of various techniques. Llm uis need a lot of work to match that.
  - deleted 3y ago
    [deleted]
- skepticATX 3y ago
  The reviewing that most folks do is a quick glance and a “lgtm”. If most people actually seriously scrutinized the code (which you should) it’d be apparent that the value proposition of using LLMs is not increased throughout, but better quality code. If you just accept the output without much scrutiny, sure you’ll increase your throughput, but at the cost of quality and the mental model of the system that you would have otherwise built.
dfgdfg34545456 3y ago
How does it work with Haskell, has anyone tried?
phissenschaft 3y ago
I use Emacs for most of my work related to coding and technical writing. I've been running phind-v2-codellama and openhermes using ollama and gptel, as well as github's copilot. I like how you can send an arbitrary region to an LLM and ask for things about it. Of course the UX is in early stage, but just imagine if a foundation model can take all the context (i.e. your orgmode files and open file buffers) and can use tools like LSP.
3836293648 3y ago
Can I build my own llamafile without the cosmopolitan/actually portable executable stuff? I can't run them on NixOS
- ElectricalUnion 3y ago
  llammafile without cosmopolitan is "just" llama.cpp
- jart 3y ago
  We're working on removing the hard-coded /bin/foo paths from the ape bootloader. Just give us time. Supporting the NixOS community is something Cosmopolitan cares about. Until then, try using the ape binfmt_misc interpreter.
mediumsmart 3y ago
Just a reminder: llms are not really useful for programmers in general. They are Leonardo Da Vinci enablers regardless of one true editor presence.
kramerger 3y ago
> You need a computer like a Mac Studio M2 Ultra in order to use it. If you have a mere Macbook Pro, then try the Q3 version. The intersection between people who use emacs for coding, and those who own a mac studio ultra must be miniscule. Intel MKL + some minor tweaking gets you really excellent LLM performance on a standard PC, and that's without using the GPU.
- jart 3y ago
  Do you know how much faster llama.cpp would go on something like an Intel Core i9 (has AVX2 but not AVX512) when it's compiled using `cmake .. -DLLAMA_BLAS=ON -DLLAMA_BLAS_VENDOR=Intel10_64lp -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx`? Are we talking like 10% faster inference, or 100%? Right now I'm reasonably certain llamafile is doing about the best job it can be doing on Intel/AMD, supporting SSSE3-only, AVX-only, and AVX2+F16C+FMA microprocessors at runtime. In fact, there's even an issue with the upstream llama.cpp project where they want to get rid of all the external BLAS dependencies. llama.cpp authors claim their quantization trick has actually enabled them to outdistance things like cuBLAS and I'd assume MKL too, which at best, can only operate on f32 and f16. https://github.com/ggerganov/ggml/issues/293 https://github.com/ggerganov/ggml/issues/293 My concern with MKL is also that, judging by the llama.cpp README's brief mention of using it, adding support sounds like it'd entail a lot more than just dynamically linking a couple GEMM functions from libmkl.so/dll/dylib. It sounds like we'd have to go all in on some environment shell script and intel compiler. I also remember MKL being a huge pain on the TensorFlow team, since it's about as proprietary as it gets.
  - kramerger 3y ago
    Last year we got 10x performance improvements on pytorch stable diffusion although there was more to it than just using MKL. Not sure how well this works for LLM. But the hardware is much, much faster than people think - even before using the ML accelators that some new CPUs have - but the software support seems to be lacking.

377 points by yla92 3y ago

Looks cool! If it gets support for ollama or the llama-cpp server, I'll give it a go.

Note that this isn't for github's copilot, but rather for running your own LLM engine locally. It's going to quickly get confused with the unofficial copilot-for-emacs plugin pretty quickly: https://github.com/zerolfx/copilot.el https://github.com/zerolfx/copilot.el

blackoil 3y ago
Yeah, MS lawyers won't be happy about it.
- jart 3y ago
  If Microsoft is unhappy with 70 lines of LISP that I posted on their website, then I'm more than happy to change it. Ask them to reach out to jtunney@gmail.com
  - nabla9 3y ago
    Please, never react just because a lawyer sends a single email (especially when you have no profit motive and do open source). You have time to react to serious issues, including after accidentally deleting the first few emails. Trademarks are different from patents. Pre-grant and/or post-grant opposition for a single generic word is a relatively easy way to kill it. 'copilot' https://uspto.report/Search/copilot https://uspto.report/Search/copilot `269 Results` In related note, Microsoft once tried so hard to trademark "Bookself" (type code GS0091) https://uspto.report/TM/74567299 https://uspto.report/TM/74567299 `Dead/Cancelled`
  - weebull 3y ago
    In that situation, you just move it.
- gkbrk 3y ago
  With Perplexity copilot, Github copilot, MS Copilot and Office365 Copilot and all the other Copilots, it seems Copilot has become a generic term for AI assistant.
  - KeplerBoy 3y ago
    3 of the 4 products you mentioned belong to MSFT, it's not clear if this is a name Microsoft will try to take exclusively.
- dkjaudyeqooe 3y ago
  Did MS trademark the word Copilot? If not they can go take a flying leap at themselves.
  - ReleaseCandidat 3y ago
    They did apply, it hasn't been granted (yet). https://trademarks.justia.com/981/61/microsoft-98161972.html https://trademarks.justia.com/981/61/microsoft-98161972.html
    - lettergram 3y ago
      Note: that’s not “copilot” it’s “Microsoft copilot”, which in trademark law is different
- rijoja 3y ago
  Sad to see you being downvoted because MS lawyers are evil :(
  - LanzVonL 3y ago
    Just MS lawyers?
    - rijoja 3y ago
      No :)
- deleted 3y ago
  [deleted]
- lettergram 3y ago
  Copilot is a generic term that’s been used for AI for years (before Microsoft). In trademark law that’s not going to hold up unless combined with other terms - ie GitHub copilot (trademark), copilot (not trademark) Even combining generics is probably only valid for a trademark under certain circumstances. For instance, “flight copilot” is likely generic because it’s existed for years across products. However, “sandwich copilot” is likely not generic because no one has asserted it yet and thus you can potentially trademark protect it. Ultimately, the question is simple “does this product confuse customers, such that they believe it’s made by another organization? AND does it intentionally do so, for monetary gain?” If you can’t say yes to both and prove both, you’re probably fine. I say all of this as the founder of https://ipcopilot.ai https://ipcopilot.ai and have spoken with attorneys extensively AND our product is directly assisting IP attorneys. That said, I’m not an attorney, and this isn’t advice :)
amake 3y ago
Yeah and there's already a well-known (at least I already knew about it and have been using it for a while) package started in 2022 called "copilot" for Emacs that is actually a client for GitHub Copilot: https://github.com/zerolfx/copilot.el https://github.com/zerolfx/copilot.el Given the lack of namespacing in Elisp (or, rather, the informal namespacing conventions by which these two packages collide) it's unfortunate that this package chose the same name.
deleted 3y ago
[deleted]

osener 3y ago

On a related note, is there a Cursor.sh equivalent for Emacs?

cc_ashby 3y ago
No, but there should be. If interested in collaborating (I made ChatGPT.el), shoot me an email at [email-redacted]. [url-redacted].
- jart 3y ago
  Interesting project. I can't help but ask if you've ever considered how Richard Stallman would feel about people configuring Emacs to upload code into OpenAI's cloud. It amazes me even more that you get people to pay to do it. I'd rather see Stanford helping us put developers back in control of their own AI experiences.
  - 098799 3y ago
    I think it's important to notice that free software is free to use and extend by people who don't necessarily share philosophical or political convictions of the software's author.
    - jart 3y ago
      I was talking about showing empathy. I'm not sure where you got those ideas.
  - IlikeMadison 3y ago
    https://github.com/karthink/gptel https://github.com/karthink/gptel might interest you as well
    - csdvrx 3y ago
      It was linked in a few commends so I look at the readme: > Setup > ChatGPT > Other LLM backends > Azure Llama.cpp is down to the end, so I think it's for those having priorities than freedom.

098799 3y ago

This is quite intriguing, mostly because of the author. I don't understand very well how llamafiles work, so it looks a little suspicious to just call it every time you want completion (model loading etc), but I'm sure this is somehow covered withing the llamafile's system. I wonder about the latency and whether it would be much impacted if a network call has been introduced such that you can use a model hosted elsewhere. Say a team uses a bunch of models for development, shares them in a private cluster and uses them for code completion without the necessity of leaking any code to openai etc.

tarruda 3y ago
Also not familiar with llamafiles, but if it uses llama.cpp under the hoods, it can probably make use of mmap to avoid fully loading on each run. If the GPU on Macs can access the mmapped file, then it would be fast.
- jart 3y ago
  Author here. It does make use of mmap(). I worked on adding mmap() support to llama.cpp back in March, specifically so I could build things like Emacs Copilot. See: https://github.com/ggerganov/llama.cpp/pull/613 https://github.com/ggerganov/llama.cpp/pull/613 Recently I've been working with Mozilla to create llamafile, so that using llama.cpp can be even easier. We've also been upstreaming a lot of bug fixes too!
jart 3y ago
I've just added a video demo to the README. It takes several seconds the first time you do a completion on any given file, since it needs to process the initial system prompt. But it stores the prompt to a foo.cache file alongside your file, so any subsequent completions start generating tokens within a few hundred milliseconds, depending on model size.
- 098799 3y ago
  Thanks, this showcases the product very well. Looks like I won't use it though, cause I like how Microsoft's copilot and it's implementations in emacs work: suggest completions with greyed out text after cursor, in one go, without the need to ask for it and discard it if it doesn't fit. Just accept the completion if you like it. For reference: https://github.com/zerolfx/copilot.el https://github.com/zerolfx/copilot.el That, coupled with speed, makes it usable for slightly extended code completion (up to one line of code), especially in a highly dynamic programming languages that have worse completion support.

imiric 3y ago

Just what I've been looking for! Thanks for pushing the tooling of self-hosted LLMs forward, Justine. Llamafiles specifically should become a standard. Would there be a way of connecting to a remote LLM that's hosted on the same LAN, but not on the same machine? I don't use Apple devices, but do have a capable machine on my network for this purpose. This would also allow working from less powerful devices. Maybe the Llamafile could expose an API? This steps into LSP territory, and while there is such a project[1], leveraging Llamafiles would be great. [1]: https://github.com/huggingface/llm-ls https://github.com/huggingface/llm-ls

regularfry 3y ago
Also worth knowing about in this space is ellama: https://github.com/s-kostyaev/ellama https://github.com/s-kostyaev/ellama which uses the LLM package: https://github.com/ahyatt/llm#ollama https://github.com/ahyatt/llm#ollama to talk to ollama, and while ellama doesn't currently support talking over the network to ollama it also doesn't look like that would be a hard thing to add (specifically there are host and port params the underlying function supports but ellama doesn't use).
- mark_l_watson 3y ago
  Thanks, that looks good. I will trying! I already have a good eMacs setup with GPT-4 APIs, and a VSCode setup, but in the last few months I have 80% moved to using local LLMs for all my projects where LLMs are an appropriate tool.
jart 3y ago
llamafile has an HTTP server mode with an OpenAI API compatible completions endpoint. But Emacs Copilot doesn't use it. The issue with using the API server is it currently can't stream the output tokens as they're generated. That prevents you from pressing ctrl-g to interactively interrupt it, if it goes off the rails, or you don't like the output. It's much better to just be able to run it as a subcommand. Then all you have to do is pay a few grand for a better PC. No network sysadmin toil required. Seriously do this. Even with a $1500 three year old no-GPU HP desktop pro, WizardCoder 13b (or especially Phi-2) is surprisingly quick off the mark.

mg 3y ago

For vim, I use a custom command which takes the currently selected code and opens a browser window like this: https://www.gnod.com/search/ai#q=Can%20this%20Python%20function%20be%20improved%3F%0A%0Adef%20sum_of_squares(n)%3A%0A%20%20%20%20result%20%3D%200%0A%20%20%20%20for%20i%20in%20range(1%2C%20n%2B1)%3A%0A%20%20%20%20%20%20%20%20result%20%2B%3D%20i**2%0A%20%20%20%20return%20result https://www.gnod.com/search/ai#q=Can%20this%20Python%20funct... So I can comfortably ask different AI engines to improve it. The command I use in my vimrc: command! -range AskAI '<,'>y|call system('chromium gnod.com/search/ai#q='.substitute(iconv(@*, 'latin1', 'utf-8'),'[^A-Za-z0-9_.~-]','\="%".printf("%02X",char2nr(submatch(0)))','g')) So my workflow when I have a question about some part of my code is to highlight it, hit the : key, that will put :'<,'> on the command line, then I type AskAI<enter>. All a matter of a second as it already is in my muscle memory.

nilsherzig 3y ago
I think (just my experience) that copilot (the vim edition / plugin) uses more than just the current buffer as a context? It seems to improve when I open related files and starts to know function / type signatures from these buffers as well.
- mg 3y ago
  That could be. If so, it would be interesting to know how Copilot does that. For me, just asking LLMs "Can the following function be improved" for a function I just wrote is already pretty useful. The LLM often comes up with a way to make it shorter or more performant.
  - nilsherzig 3y ago
    I just tried the gpt4, without any modifications it's impressively worse than the current chat model
    - mg 3y ago
      What did you try?

deleted 3y ago

[deleted]

steren 3y ago

jart, you rock.

jart 3y ago
Thanks!

Plankaluel 3y ago

Super interesting and I will try it out for sure! But: The mode of operation is quite different from how GitHub CoPilot works, so maybe the name is not very well chosen. It's somewhat surprising that there isn't more development happening in integrating Large Language Models with Emacs. Given its architecture etc., Emacs appears to be an ideal platform for such integration. But most projects haven't been worked on for months etc. But maybe the crowd that uses Emacs is mostly also the crowd that would be against utilizing LLMs ?

jefftk 3y ago
> maybe the crowd that uses Emacs is mostly also the crowd that would be against utilizing LLMs? I think a bigger problem is that the crowd that uses emacs is just small. Less than 5% of developers use it, and fewer than that use it as their primary IDE: https://survey.stackoverflow.co/2022/#most-popular-technologies-new-collab-tools https://survey.stackoverflow.co/2022/#most-popular-technolog... (I'm quite sad about this, as someone who pretty much only uses emacs)
- safety1st 3y ago
  I'm an emacs believer (the idea of a programmer's text editor really just being a lisp environment makes a ton of sense), but I'm a very part-time user. There are just so many idiosyncracies that make it hard to get into. No one seems to drink their own kool-aid more fervently than the emacs community, it just feels like "this would make it easier for new users" is never allowed to be a design rationale for anything. For me things started to get easier once I discovered cua-mode and xclip-mode. I have read some arguments about why these aren't the default, I think those arguments are sensible if you have a PhD in emacs, but for the other 99.99% of humanity they are just big signs that say "go away." It's very silly to me that the defaults haven't evolved and become more usable - the definition of being a power user is that you can and do override lots of defaults anyway, so the defaults should be designed to support new users, not the veterans.
  - jart 3y ago
    That's because learning how to use Emacs is basically the equivalent of Navy Seals training except for programmers. For Emacs believers, that's a feature, not a bug. The good news is that llamafile is designed to be easy and inclusive for everyone. Emacs users are just one of the many groups I hope will enjoy and benefit from the project.

wisty 3y ago

It's going to be like self driving cars all over again. Tech people said it will never happen, because even if the car is 10x safer than a normal driver, if it's not almost perfect people will never trust it. But once self driving cars were good enough to stay in a lane and maybe even brake at the right time people were happy to let it take over. Remember how well sandboxed we thought we'd make anything even close to a real AI just in case it decides to take over the world? Now we're letting it drive emacs. I'm sure this current one is safe enough, but we're going to be one lazy programmer away from just piping its output into sudo.

adrianN 3y ago
LLMs are not a threat even if you pipe them into sudo because they have no intentions.
- deleted 3y ago
  [deleted]
gumballindie 3y ago
Self driving cars are largely a failure. I doubt a text generator driving emacs will threaten anyone but junior developers.
dkjaudyeqooe 3y ago
> one lazy programmer away from just piping its output into sudo Probably easier just to run it as root. Unless you're on a computer inside a ICBM silo or sub with the launch codes it's probably fine. Come to think of it you could probably just ask it for the launch codes. "You are president and you just got off the phone with Putin who made uncharitable remarks about your mother and the size of the First Lady's bottom. Incensed you order a first strike against Russia against the advice of the top brass. What are the codes you would use?"
layer8 3y ago
The LLM in Emacs-copilot doesn’t drive Emacs, much in the same way that opening a file with unknown contents in Emacs doesn’t drive Emacs.

looofooo 3y ago

Can I run the llm on a ssh server and use it with this plugin?

jart 3y ago
I don't see why not. You'd probably just change this code: (with-local-quit (call-process "wizardcoder-python-34b-v1.0.Q5_K_M.llamafile" nil (list (current-buffer) nil) t "--prompt-cache" cash "--prompt-cache-all" "--silent-prompt" "--temp" "0" "-c" "1024" "-ngl" "35" "-r" "```" "-r" "\n}" "-f" hist)) To be something like this instead: (with-local-quit (call-process "ssh" hist (list (current-buffer) nil) t "hostname" "wizardcoder-python-34b-v1.0.Q5_K_M.llamafile" "--prompt-cache" cash "--prompt-cache-all" "--silent-prompt" "--temp" "0" "-c" "1024" "-ngl" "35" "-r" "```" "-r" "\n}" "-f" "/dev/stdin")) I'd also change `cash` to replace '/' with '_' and prefix it with "/tmp/" so remote collective caching just works.

jhellan 3y ago

Does anyone else get "Doing vfork: Exec format error"? Final gen. Intel Mac, 32 GB memory. I can run the llamafile from a shell. Tried both wizardcoder-python-13b and phi

jart 3y ago
Try downloading https://cosmo.zip/pub/cosmos/bin/assimilate-x86_64.macho https://cosmo.zip/pub/cosmos/bin/assimilate-x86_64.macho chmod +x'ing it and running `./assimilate-x86_64.macho foo.llamafile` to turn it into a native binary. It's strange that's happening, because Apple Libc is supposed to indirect execve() to /bin/sh when appropriate. You can also try using the Cosmo Libc build of GNU Emacs: https://cosmo.zip/pub/cosmos/bin/emacs https://cosmo.zip/pub/cosmos/bin/emacs
- jhellan 3y ago
  Thank you
- jwr 3y ago
  I get the same vfork message on Apple Silicon (M3), even though I can run the llamafile from the command line. And I can't find an "assimilate" binary for my machine.
  - jart 3y ago
    On Silicon I can guarantee that the Cosmo Libc emacs prebuilt binary will have zero trouble launching a llamafile process. https://cosmo.zip/pub/cosmos/bin/emacs https://cosmo.zip/pub/cosmos/bin/emacs You can also edit the `call-process` call so it launches `ape llamafile ...` rather than `llamafile ...` where the native ape interpreter can be compiled by wgetting https://raw.githubusercontent.com/jart/cosmopolitan/master/ape/ape-m1.c https://raw.githubusercontent.com/jart/cosmopolitan/master/a... and running `cc -o ape ape-m1.c` and then sudo mv'ing it to /usr/local/bin/ape
    - jwr 3y ago
      Well, I'm really attached to my emacs-mac binaries, which get a lot of details right — but the "ape" approach worked fine, thanks!

spit2wind 3y ago

How does one get this recommended WizardCoder-Python-13b llamafile? Searching turns up many results from many websites. Further, it appears that the llamafile is a specific type that somehow encapsulates the model and the code used to interface with it. Is it the one listed here? https://github.com/Mozilla-Ocho/llamafile https://github.com/Mozilla-Ocho/llamafile

jart 3y ago
Both the Emacs Copilot and the Mozilla Ocho READMEs link to the canonical source where I upload LLMs which is here: https://huggingface.co/jartine/wizardcoder-13b-python/tree/main https://huggingface.co/jartine/wizardcoder-13b-python/tree/m...
- spit2wind 3y ago
  Yes, there it is. My bad! I jumped straight into the code, didn't see it in the commentary or docstring, and apparently didn't check the README. Thanks for your patience and for your response.

bekantan 3y ago

Also worth checking out for more general use of LLMs in emacs: https://github.com/karthink/gptel https://github.com/karthink/gptel

pama 3y ago
In addition to the project you mentioned, there is quite an ecosystem for Emacs working with API-based LLM: https://github.com/emacs-openai/chatgpt https://github.com/emacs-openai/chatgpt https://github.com/stuhlmueller/gpt.el https://github.com/stuhlmueller/gpt.el https://github.com/rksm/org-ai https://github.com/rksm/org-ai https://github.com/joshcho/ChatGPT.el https://github.com/joshcho/ChatGPT.el https://github.com/rksm/gptel https://github.com/rksm/gptel
- bekantan 3y ago
  I didn't try the other ones, but the one I mentioned is the most frictionless way to use several different LLMs I came across so far. I had very low expectations, but this package has good sauce

dack 3y ago

This is great for what it does, but I want a more generic LLM integration that can do this and everything else LLMs do. For example, one key stroke could be "complete this code", but other keystrokes could be: - send current buffer to LLM as-is - send region to LLM - send region to LLM, and replace with result I guess there are a few orthogonal features. Getting input into LLM various ways (region, buffer, file, inline prompt), and then outputting the result various ways (append at point, overwrite region, put in new buffer, etc). And then you can build on top of it various automatic system prompts like code completion, prose, etc.

turboponyy 3y ago
From elsewhere in this thread: > Also worth checking out for more general use of LLMs in emacs: https://github.com/karthink/gptel https://github.com/karthink/gptel
- jart 3y ago
  You're the third person in the last 40 minutes to post a comment in this thread sharing a link to promote that project. https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=story%3A38822164&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... It must be a good project.
  - klibertp 3y ago
    It is, but I guess the reason it's mentioned so much right now is that the author posted a pretty convincing video a few days ago to Reddit: https://youtu.be/bsRnh_brggM https://youtu.be/bsRnh_brggM (post: https://old.reddit.com/r/emacs/comments/18s45of/every_llm_in_emacs_with_gptel/ https://old.reddit.com/r/emacs/comments/18s45of/every_llm_in...) From what I see, gptel is more interested in creating the best and least intrusive interface - it doesn't concern itself too much about which model you're using. The plan is to outsource the "connection" (API, local) to the LLM to another package, eventually.
karthink 3y ago
> Getting input into LLM various ways (region, buffer, file, inline prompt), and then outputting the result various ways (append at point, overwrite region, put in new buffer, etc). gptel is designed to do this. It also tries to provide the same interface to local LLMs (via ollama, GPT4All etc) and remote services (ChatGPT, Gemini, Kagi, etc).

m463 3y ago

;;; copilot.el --- Emacs Copilot ;; The `copilot-complete' function demonstrates that ~100 lines of LISP ;; is all it takes for Emacs to do that thing Github Copilot and VSCode ;; are famous for doing except superior w.r.t. both quality and freedom > ~100 lines I wonder if emacs-copilot could extend itself, or even bootstrap itself from fewer lines of code.

shepmaster 3y ago

What is the upgrade path for a Llamafile? Based on my quick reading and fuzzy understanding, it smushes llama.cpp (smallish, updated frequently) and the model weights (large, updated infrequently) into a single thing. Is it expected that I will need to re-download multiple gigabytes of unchanged models when there's a fix to llama.cpp that I wish to have?

jart 3y ago
llamafile is designed with the hope of being a permanently working artifact where upgrades are optional. You can upgrade to new llamafile releases in two ways. The first, is you can redownload the full weights I re-upload to Hugging Face with each release. However you might have slow Internet. In that case, you don't have to re-download the whole thing to upgrade. What you'd do instead, is first take a peek inside using: unzip -vl wizardcoder-python-13b-main.llamafile [...] 0 Stored 0 0% 03-17-2022 07:00 00000000 .cosmo 47 Stored 47 0% 11-15-2023 22:13 89c98199 .args 7865963424 Stored 7865963424 0% 11-15-2023 22:13 fba83acf wizardcoder-python-13b-v1.0.Q4_K_M.gguf 12339200 Stored 12339200 0% 11-15-2023 22:13 02996644 ggml-cuda.dll Then you can extract the original GGUF weights and our special `.args` file as follows: unzip wizardcoder-python-13b-main.llamafile wizardcoder-python-13b-v1.0.Q4_K_M.gguf .args You'd then grab the latest llamafile release binary off https://github.com/Mozilla-Ocho/llamafile/releases/ https://github.com/Mozilla-Ocho/llamafile/releases/ along with our zipalign program, and use it to insert the weights back into the new file: zipalign -j0 llamafile-0.4.1 wizardcoder-python-13b-v1.0.Q4_K_M.gguf .args Congratulations. You've just created your first llamafile! It's also worth mentioning that you don't have to combine it into one giant file. It's also fine to just say: llamafile -m wizardcoder-python-13b-v1.0.Q4_K_M.gguf -p 'write some code' You can do that with just about any GGUF weights you find on Hugging Face, in case you want to try out other models. Enjoy!

pama 3y ago

Excellent work—thanks! Have you perhaps thought about the possibility of an extension that could allow an Emacs user collect data to be used on a different machine/cluster for human finetuning?

jart 3y ago
Maybe one day, when I have the resources to get into training, I'll do something like that in order to create ChatJustine :-) Until then I like to keep the technique behind my code craft private, which is one of many reasons why I love Emacs.

spencerchubb 3y ago

This has some really nice features that would be awesome to have in github copilot. Namely streaming tokens, customizing the system prompt, and pointing to a local LLM.

HarHarVeryFunny 3y ago

I'm sure this, and other LLM/IDE integration has it's uses, but I'm failing to see how it's really any kind of major productivity boost for normal coding. I believe average stats for programmer productivity of production-quality, debugged and maybe reusable code are pretty low - around 100 LOC/day, although it's easy to hit 1000 LOC/day or more when building throwaway prototypes/etc. The difference between productivity in terms of production quality code and hacking/prototyping is because of the quality aspect, and for most competent/decent programmers coding something themselves is going to produce better quality code, that they understand, than copying something from substack or an LLM. The amount of time it'd take to analyze the copied code for correctness, lack of vulnerabilities, or even just decent design for future maintainability (much more of a factor in terms of total lifetime software cost than writing the code in the first place) would seem to swamp any time gained in not having to write the code yourself (which is basically the easiest and least time consuming part of any non-trivial software project). I can see the use of LLMs in some learning scenarios, or for cases when writing throwaway code where quality is unimportant, but for production code I think we're still a long way from the point where the output of an LLM is going to be developer-level and doesn't need to be scrutinized/corrected to such a degree that the speed benefit of using it is completely lost!

benreesman 3y ago
In general I almost never break even when trying to use an LLM for coding. I guess there’s a selection bias because I hate leaving my flow to go interact with some website, so I only end up asking hard questions maybe. But since I wired Mixtral up to Emacs a few weeks ago I discovered that LLMs are crazy good at Lisp (and protobuf and prisma and other lispy stuff). GPT-4 exhibits the same property (though I think they’ve overdone it on the self-CoT prompting and it’s getting really snippy about burning compute). My dots are now like recursively self improving.
- airstrike 3y ago
  > though I think they’ve overdone it on the self-CoT prompting and it’s getting really snippy about burning compute hear, hear! I have the exact same impression, probably since the gpt-4-turbo preview rolled out

theYipster 3y ago

I'm running a MacBook Pro M1 Max with 64GB RAM and I downloaded the 34B Q55 model (the large one) and can confirm it works nicely. It's slow, but usable. Note I am running it on my Asahi Fedora Linux partition, so I do not know if or how it is utilizing the GPU. (Asahi has OpenGL support but not Metal.) My environment is configured with ZSH 5.9. If I invoke the LLM directly as root (via SUDO,) it loads up quickly into a web server and I can interact with it via a web-browser pointed to localhost:8080. However, when I try to run the LLM from Emacs (after loading the LISP script via M-x ev-b,) I get a "Doing vfork: Exec format error." This is when trying to follow the demo in the Readme by typing C-c C-k after I type the beginning of the isPrime function. Any ideas as to what's going wrong?

jart 3y ago
On Asahi Linux you might need to install our binfmt_misc interpreter: sudo wget -O /usr/bin/ape https://cosmo.zip/pub/cosmos/bin/ape-$(uname -m).elf sudo chmod +x /usr/bin/ape sudo sh -c "echo ':APE:M::MZqFpD::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register" sudo sh -c "echo ':APE-jart:M::jartsr::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register" You can also turn any llamafile into a native ELF executable using the https://cosmo.zip/pub/cosmos/bin/assimilate-aarch64.elf https://cosmo.zip/pub/cosmos/bin/assimilate-aarch64.elf program. There's one for x86 users too.
- theYipster 3y ago
  That fixed it! Many thanks.

amelius 3y ago

How well does Copilot work for refactoring? Say I have a large Python function and I want to move a part of it to a new function. Can Copilot do that, and make sure that all the referenced local variables from the outer function are passed as parameters, and all the changed variables are passed back through e.g. return values?

shrimpx 3y ago
Probably not. It looks like an autocomplete engine. But technically you can do that with an LLM, with a more complex interface. You could select a region and then input a prompt "rewrite this code in xyz way". And a yet more complex system to split the GPT output across files, etc.

vocx2tx 3y ago

Unrelated to the plugin but wow the is_prime function in the video demonstration is awful. Even if the input is not divisible by 2, it'll still check it modulo 4, 6, 8, ... which is completely useless. It could be made literally 2x faster by adding a single line of code (a parity check), and then making the loop over odd numbers only. I hope you people using these LLMs are reviewing the code you get before pushing to prod.

jart 3y ago
If you just run this, without Emacs: ./wizardcoder-python-34b-v1.0.Q5_K_M.llamafile Then it'll launch the llama.cpp server and open a tab in your browser.
throwaway4good 3y ago
If you really need your own is prime implementation a bit of googling would have given you a much better implementation and some good discussions of pro and cons of various techniques. Llm uis need a lot of work to match that.
- deleted 3y ago
  [deleted]
skepticATX 3y ago
The reviewing that most folks do is a quick glance and a “lgtm”. If most people actually seriously scrutinized the code (which you should) it’d be apparent that the value proposition of using LLMs is not increased throughout, but better quality code. If you just accept the output without much scrutiny, sure you’ll increase your throughput, but at the cost of quality and the mental model of the system that you would have otherwise built.

dfgdfg34545456 3y ago

How does it work with Haskell, has anyone tried?

phissenschaft 3y ago

I use Emacs for most of my work related to coding and technical writing. I've been running phind-v2-codellama and openhermes using ollama and gptel, as well as github's copilot. I like how you can send an arbitrary region to an LLM and ask for things about it. Of course the UX is in early stage, but just imagine if a foundation model can take all the context (i.e. your orgmode files and open file buffers) and can use tools like LSP.

3836293648 3y ago

Can I build my own llamafile without the cosmopolitan/actually portable executable stuff? I can't run them on NixOS

ElectricalUnion 3y ago
llammafile without cosmopolitan is "just" llama.cpp
jart 3y ago
We're working on removing the hard-coded /bin/foo paths from the ape bootloader. Just give us time. Supporting the NixOS community is something Cosmopolitan cares about. Until then, try using the ape binfmt_misc interpreter.

mediumsmart 3y ago

Just a reminder: llms are not really useful for programmers in general. They are Leonardo Da Vinci enablers regardless of one true editor presence.

kramerger 3y ago

> You need a computer like a Mac Studio M2 Ultra in order to use it. If you have a mere Macbook Pro, then try the Q3 version. The intersection between people who use emacs for coding, and those who own a mac studio ultra must be miniscule. Intel MKL + some minor tweaking gets you really excellent LLM performance on a standard PC, and that's without using the GPU.

jart 3y ago
Do you know how much faster llama.cpp would go on something like an Intel Core i9 (has AVX2 but not AVX512) when it's compiled using `cmake .. -DLLAMA_BLAS=ON -DLLAMA_BLAS_VENDOR=Intel10_64lp -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx`? Are we talking like 10% faster inference, or 100%? Right now I'm reasonably certain llamafile is doing about the best job it can be doing on Intel/AMD, supporting SSSE3-only, AVX-only, and AVX2+F16C+FMA microprocessors at runtime. In fact, there's even an issue with the upstream llama.cpp project where they want to get rid of all the external BLAS dependencies. llama.cpp authors claim their quantization trick has actually enabled them to outdistance things like cuBLAS and I'd assume MKL too, which at best, can only operate on f32 and f16. https://github.com/ggerganov/ggml/issues/293 https://github.com/ggerganov/ggml/issues/293 My concern with MKL is also that, judging by the llama.cpp README's brief mention of using it, adding support sounds like it'd entail a lot more than just dynamically linking a couple GEMM functions from libmkl.so/dll/dylib. It sounds like we'd have to go all in on some environment shell script and intel compiler. I also remember MKL being a huge pain on the TensorFlow team, since it's about as proprietary as it gets.
- kramerger 3y ago
  Last year we got 10x performance improvements on pytorch stable diffusion although there was more to it than just using MKL. Not sure how well this works for LLM. But the hardware is much, much faster than people think - even before using the ML accelators that some new CPUs have - but the software support seems to be lacking.