8 ms·

Llama.vim – Local LLM-assisted text completion

530 points by kgwgk 2y ago

behnamoh 2y ago
[flagged]
- halyconWays 2y ago
  [flagged]
- dang 2y ago
  Please make your substantive points without putting down others, and please omit such complaints about downvotes. The latter are inevitably unsubstantiated (since vote data isn't public), usually imaginary, and often provocative. If you're worried about abuse, the thing to do is email hn@ycombinator.com so we can take a look. All this is in HN's guidelines: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html. If you'd please review those and stick to them, we'd appreciate it.
  - halyconWays 2y ago
    Why do you consider it "putting down others" to say that they're wrappers of other people's work, which they are?
- ninth_ant 2y ago
  The list of things people who deserves more credit is endless, for all intents and purposes. People make kernels, drivers, file system, drivers, languages, compilers, container systems, security infrastructure, and the list goes on. It’s literally an xkcd comic. Instead of suggesting ollama folks get too much credit, perhaps simply acknowledge the people you feel are undervalued. Because by packaging together the various bits into an easy to use system with good documentation, the ollama folks have done a lot a good for me and any attention and credit they get is richly deserved.
  - IAmNotACellist 2y ago
    It's not that they get too much credit (people are excited about this tech and there will be frontrunners and winners), it's that they're a wrapper for llama.cpp and give no acknowledgement or credit to the FOSS that makes their project work.
dingnuts 2y ago
Is anyone actually getting value out of these models? I wired one up to Emacs and the local models all produce a huge volume of garbage output. Occasionally I find a hosted LLM useful but I haven't found any output from the models I can run in Ollama on my gaming PC to be useful. It's all plausible-looking but incorrect. I feel like I'm taking crazy pills when I read about others' experiences. Surely I am not alone?
- remexre 2y ago
  I work on compilers. A friend of mine works on webapps. I've seen Cursor give him lots of useful code, but it's never been particularly useful on any of the code of mine that I've tried it on. It seems very logical to me that there'd be orders of magnitude more training data for some domains than others, and that existing models' skill is not evenly distributed cross-domain.
  - dkga 2y ago
    This. Also across languages. For example, I suppose there is a lot more content in python and javascript than Apple script, for example. (And to be fair not a lot of the python suggestions I receive are actually mindblowing good)
  - q0uaur 2y ago
    i'm still patiently waiting for an easy way to point a model at some documentation, and make it actually use that. My usecase is gdscript for godot games, and all the models i've tried so far use godot 2 stuff that's just not around anymore, even if you tell it to use godot 4 it gives way too much wrong output to be useful. I wish i could just point it at the latest godot docs and have it give up to date answers. but seeing as that's still not a thing, i guess it's more complicated than i expect.
    - doctoboggan 2y ago
      It's definitely a thing already. Look up "RAG" (Retrieval Augmented Generation). Most of the popular closed source companies offer RAG services via their APIs, and you can also do it with local llms using open-webui and probably many other local UIs.
jerpint 2y ago
It’s funny because I actually use vim mostly when I don’t want LLM assisted code. Sometimes it just gets in the way. If I do, I load up cursor with vim bindings.
- renewiltord 2y ago
  Funny. I think the most common usage I have is using it at the command line to write commands with vim set as my EDITOR so the AI completion really helps. This will help for offline support (on planes and such).
  - qup 2y ago
    Can you more specifically talk about how you use this, like with a small example?
    - renewiltord 2y ago
      Yes, I mentioned it here first and haven't changed it since (Twitter link and links include video of use) https://news.ycombinator.com/item?id=34769611 https://news.ycombinator.com/item?id=34769611 Which leads (used to lead?) here https://wiki.roshangeorge.dev/index.php/AI_Completion_In_The_Shell https://wiki.roshangeorge.dev/index.php/AI_Completion_In_The...
    - VMG 2y ago
      - composing a commit message - anything bash script related
- rapind 2y ago
  What I'd really like is a non-intrusive LLM running that I can put on a third monitor which sees what I'm working on, while I work, and tries to mirror and make suggestions WITHOUT interjecting itself into my editor. Then I just want a mappable prompt command (like <leader>P) where I can provide feedback / suggestions (or maybe I just do that via mic). This way I can choose to just ignore it for the most part, but if I see something interesting, I can refine it, prompt, cherry pick, etc. I really don't want autocomplete for anything.
ggerganov 2y ago
Hi HN, happy to see this here! I highly recommend to take a look at the technical details of the server implementation that enables large context usage with this plugin - I think it is interesting and has some cool ideas [0]. Also, the same plugin is available for VS Code [1]. Let me know if you have any questions about the plugin - happy to explain. Btw, the performance has improved compared to what is seen in the README videos thanks to client-side caching. [0] - https://github.com/ggerganov/llama.cpp/pull/9787 https://github.com/ggerganov/llama.cpp/pull/9787 [1] - https://github.com/ggml-org/llama.vscode https://github.com/ggml-org/llama.vscode
- jerpint 2y ago
  Thank you for all of your incredible contributions!
- amrrs 2y ago
  For those who don't know, He is the gg of `gguf`. Thank you for all your contributions! Literally the core of Ollama, LMStudio, Jan and multiple other apps!
  - sergiotapia 2y ago
    well hot damn! killing it!
  - halyconWays 2y ago
    [flagged]
    - madeforhnyo 2y ago
      Someone did? Could you pls share a link?
    - kamranjon 2y ago
      They collaborate together! Her name is Justine Tunney - she took her “execute everywhere” work with Cosmopolitan to make Llamafile using the llama.cpp work that Giorgi has done.
2099miles 2y ago
[flagged]
- shawnlower2 2y ago
  Uganda's about to get $10 billion dollars!
msoloviev 2y ago
I wonder how the "ring context" works under the hood. I have previously had (and recently messed around with again) a somewhat similar project designed for a more toy/exploratory setting (https://github.com/blackhole89/autopen https://github.com/blackhole89/autopen - demo video at https://www.youtube.com/watch?v=1O1T2q2t7i4 https://www.youtube.com/watch?v=1O1T2q2t7i4), and one of the main problems to address definitively is the question of how to manage your KV cache cleverly so you don't have to constantly perform too much expensive recomputation whenever the buffer undergoes local changes. The solution I came up with involved maintaining a tree of tokens branching whenever an alternative next token was explored, with full LLM state snapshots at fixed depth intervals so that the buffer would only have to be "replayed" for a few tokens when something changed. I wonder if there are some mathematical properties of how the important parts of the state (really, the KV cache, which can be thought of as a partial precomputation of the operation that one LLM iteration performs on the context) work that could have made this more efficient, like to avoid saving full snapshots or perhaps to be able to prune the "oldest" tokens out of a state efficiently. (edit: Georgi's comment that beat me by 3 minutes appears to be pointing at information that would go some way to answer my questions!)
deleted 2y ago
[deleted]
entelechy0 2y ago
I use this on-and-off again. It is nice that I can flip between this and Copilot by commenting out one line in my init.lua
eigenvalue 2y ago
This guy is a national treasure and has contributed so much value to the open source AI ecosystem. I hope he’s able to attract enough funding to continue making software like this and releasing it as true “no strings attached” open source.
- frankfrank13 2y ago
  Hard agree. This alone replaces GH Copilot/Cursor ($10+ a month)
- nacs 2y ago
  > This guy is a national treasure Agreed but he's an international treasure (his Github profile states Bulgaria).
- feznyng 2y ago
  They have: https://ggml.ai/ https://ggml.ai/ under the Company heading.
  - cosmojg 2y ago
    Georgi Gerganov is the "gg" in "ggml"
    - acters 2y ago
      Also the gg in gguf
frankfrank13 2y ago
Is this more or less the same as your VSCode version? (https://github.com/ggml-org/llama.vscode https://github.com/ggml-org/llama.vscode)
mohsen1 2y ago
Terminal coding FTW! And when you're really stuck you can use DeepSeek R1 for a deeper analysis in your terminal using `askds` https://github.com/bodo-run/askds https://github.com/bodo-run/askds
opk 2y ago
Has anyone actually got this llama stuff to be usable on even moderate hardware? I find it just crashes because it doesn't find enough RAM. I've got 2G of VRAM on an AMD graphics card and 16G of system RAM and that doesn't seem to be enough. The impression I got from reading up was that it worked for most Apple stuff because the memory is unified and other than that, you need very expensive Nvidia GPUs with lots of VRAM. Are there any affordable options?
- basilgohar 2y ago
  I can run 7B models with Q4 quantization on a 7000 series AMD APU without GPU acceleration quite acceptably fast. This is with DDR5600 RAM which is the current roadblock for performance. Larger models work but slow down. I do have 64GB of RAM but I think 32 could work. 16GB is pushing is, but should be possible if you don't have anything else open. Memory requirements depend on numerous factors. 2GB VRAM is not enough for most GenAI stuff today.
- zamadatix 2y ago
  2G is pretty low and the sizes things you can get to run fast on that set up probably aren't particularly attractive. "moderate hardware" varies but you can grab a 12 GB RTX 3060 on ebay for ~$200. You can get a lot more RAM for $200 but it'll be so slow compare the the GPU I'm not sure I'd recommend it if you actually want to use things like this interactively. If "moderate hardware" is your average office PC then it's unlikely to be very usable. Anyone with a gaming GPU from the last several years should be workable though.
  - horsawlarway 2y ago
    I'll second this, actually - $250 for a 12gb rtx 3060 is probably a better buy than $400 for 2xp40s for 16gb. It'd been a minute since I checked refurb prices and $250 for the rtx 3060 12gb is a good price. Easier on the rest of the system than a 2x card setup, and is probably a drop in replacement.
- bhelkey 2y ago
  Have you tried Ollama [1]? You should be able to run a 8b model in RAM and a 1b model in VRAM. [1] https://news.ycombinator.com/item?id=42069453 https://news.ycombinator.com/item?id=42069453
colordrops 2y ago
I've seen several posts and projects like this. Is there a summary/comparison somewhere of the various ways of running local completion/copilot?
h14h 2y ago
A little bit of a tangent, but I'm really curious what benefits could come from integrating these LLM tools more closely with data from LSPs, compilers, and other static analysis tools. Intuitively, it seems like you could provide much more context and better output as a result. Even better would be if you could fine-tune LLMs on a per-language basis and ship them alongside typical editor tooling. A problem I see w/ these AI tools is that they work much better with old, popular languages, and I worry that this will grow as a significant factor when choosing a language. Anecdotally, I see far better results when using TypeScript than Gleam, for example. It would be very cool to be able to install a Gleam-specific model that could be fed data from the LSP and compiler, and wouldn't constantly hallucinate invalid syntax. I also wonder if, with additional context & fine-tuning, you could make these models smaller and more feasible to run locally on modest hardware.
- sdesol 2y ago
  > work much better with old, popular languages I think this will improve significantly over time when when hardware becomes cheaper. As long as newer languages can map to older languages (syntax/function wise), we should be able generate enough synthetic data to make working with less known languages easier.
- rabiescow 2y ago
  you are free to contribute your own gleam llms. They're only as good as their inputs, thus if there's very few publicly available packages for a certain language that's all the input they got...
amelius 2y ago
This looks very interesting. Can this be trained on the user's codebase, or is the idea that everything must fit inside the context buffer?
- larodi 2y ago
  Models don’t get trained on your codebase in scenarios like Continue and Copilot assisted completion. It’s RAG at best and it depends on how the RAG is built. So all depends on how it fits in context and it will for the foreseeable time be like it as training is absolutely expensive compared to RAG.
binary132 2y ago
I am curious to see what will be possible with consumer grade hardware and more improvements to quantization over the next decade. Right now, even a 24GB gpu with the best models isn’t able to match the barely acceptable performance of hosted services I’m not willing to even pay $20 a month for.
morcus 2y ago
Looking for advice from someone who knows about the space - Suppose I'm willing to go out and buy a card for this purpose, what's a modestly priced graphics card with which I can get somewhat usable results running local LLM?
- estreeper 2y ago
  This is a tough question to answer, because it depends a lot on what you want to do! One way to approach it may be to look at what models you want to run and check the amount of VRAM they need. A back-of-the-napkin method taken from here[0] is: VRAM (GB) = 1.2 * number of parameters (in billions) * bits per parameter / 8 The 1.2 is just an estimation factor to account for the VRAM needed for things that aren't model parameters. Because quantization is often nearly free in terms of output quality, you should usually look for quantized versions. For example, Llama 3.2 uses 16-bit parameters but has a 4-bit quantized version, and looking at the formula above you can see that will allow you to run a 4x larger model. Having enough VRAM will allow you to run a model, but performance is dependent on a lot of other factors. For a much deeper dive into how all of this works along with price/dollar recommendations (though from last year!), Tim Dettmers wrote this excellent article: https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/ https://timdettmers.com/2023/01/30/which-gpu-for-deep-learni... Worth mentioning for the benefit of those who don't want to buy a GPU: there are also models which have been converted to run on CPU. [0] https://blog.runpod.io/understanding-vram-and-how-much-your-llm-needs/ https://blog.runpod.io/understanding-vram-and-how-much-your-...
  - morcus 2y ago
    Thank you for the background!
- loudmax 2y ago
  The bottleneck for running LLMs on consumer grade equipment is the amount of VRAM your GPU has. VRAM is RAM that's physically built into the unit and it has much higher memory bandwidth than regular system RAM. Obviously, newer GPUs will run faster than older GPUs, but you need more VRAM to be able to run larger models. A small LLM that fits into an RTX 4060's 8GB of VRAM will run faster there than it would on an older RTX 3090. But the 3090 has 24GB of VRAM, so it can run larger LLMs that the 4060 simply can't handle. Llama.cpp can split your LLM onto multiple GPUs, or split part of it onto the CPU using system RAM, though that last option is much much slower. The more of the model you can fit into VRAM, the better. The Apple M-series macbooks have unified memory, so the GPU has higher bandwidth access to system RAM than would be available over a PCIe card. They're not as powerful as Nvidia GPUs, but they're a reasonable option for running larger LLMs. It's also worth considering AMD and Intel GPUs, but most of the development in the ML space is happening on Nvidia's CUDA architecture, so bleeding edge stuff tends to be Nvidia first and other architectures later, if at all.
awwaiid 2y ago
The blinking cursor in demo videos is giving me heart palpitations! But this is super cool. It makes me wonder how Linux is doing on M* hardware.
- amelius 2y ago
  > It makes me wonder how Linux is doing on M* hardware. Not so great. Zero legal status. No blessing by Apple. Can be taken down whenever Apple execs decide to.
cfiggers 2y ago
Do people with "Copilot+ PCs" get benefits running stuff like this from the much-vaunted AI coprocessors in for e.g. Snapdragon X Elite chips?
estreeper 2y ago
Very exciting - I'm a long-time vim user but most of my coworkers use VSCode, and I've been wanting to try out in-editor completion tools like this. After using it for a couple hours (on Elixir code) with Qwen2.5-Coder-3B and no attempts to customize it, this checks a lot of boxes for me: - I pretty much want fancy autocomplete: filling in obvious things and saving my fingers the work, and these suggestions are pretty good - the default keybindings work for me, I like that I can keep current line or multi-line suggestions - no concerns around sending code off to a third-party - works offline when I'm traveling - it's fast! So I don't need to remember how to run the server, I'll probably set up a script to check if it's running and if not start it up in the background and run vim, and alias vim to use that. I looked in the help documents but didn't see a way to disable the "stats" text after the suggestions, though I'm not sure it will bother me that much.
- ggerganov 2y ago
  Appreciate the feedback! Currently, there isn't a user-friendly way to disable the stats from showing apart from modifying the "'show_info': 0" value directly in the plugin implementation. These things will be improved with time and will become more user-friendly. A few extra optimizations will soon land which will further improve the experience: - Speculative FIM - Multiple suggestions
  - tomnipotent 2y ago
    First extension I've used that perfectly autocompletes Go method receivers. First tab completes just "func (t *Type)" so then I can type the first few characters of something I'm specifically looking for or wait for the first recommendation to kick in. I hope this isn't just a coincidence from the combination of model and settings...
- douglee650 2y ago
  So I assume you have tried vscode vim mode. Would love to hear your thoughts. Are you on Mac/Linux or Windows?
mrinterweb 2y ago
Been using this for a couple hours, and this is really nice. It is a great alternative to something like Github Copilot. Appreciate how simple and fast this is.
mijoharas 2y ago
Can anyone compare this to Tabbyml?[0] I just set that up yesterday for emacs to check it out. The context gathering seems very interesting[1], and very vim-integrated, so I'm guessing there isn't anything very similar for Tabby. I skimmed the docs and saw some stuff about context for the Tabby chat feature[2] which I'm not super interested in using even if the docs adding sounds nice, but nothing obvious for the auto completion[3]. Does anyone have more insight or info to compare the two? As a note, I quite like that the LLM context here "follows" what you're doing. It seems like a nice idea. Does anyone know if anyone else does something similar? [0] https://www.tabbyml.com/ https://www.tabbyml.com/ [1] https://github.com/ggerganov/llama.cpp/pull/9787#issue-2572915687 https://github.com/ggerganov/llama.cpp/pull/9787#issue-25729... "global context onwards" [2] https://tabby.tabbyml.com/docs/administration/context/ https://tabby.tabbyml.com/docs/administration/context/ [3] https://tabby.tabbyml.com/docs/administration/code-completion/ https://tabby.tabbyml.com/docs/administration/code-completio...
- ghthor 2y ago
  I’ve been using Tabby happily since May 2023
  - mijoharas 2y ago
    Have you setup the context providers, and do you find that important for getting good completions?
- mijoharas 2y ago
  Ahhh, it seems that tabby does use RAG and context providers for the code completion. Interesting: > During LLM inference, this context is utilized for code completion Hmmm... I wonder what's better. As I'm coding I jump and search to relevant parts of the codebase to build up my own context for solving the problem, and I expect that's likely better than RAG. Llama.vim seems to follow this model, while tabby could theoretically get at things I'm not looking at/haven't looked at recently...
s-skl 2y ago
Really awesome work! Do anyone know what's the tool/terminal configuration he's using on the video demo to embed CPU/GPU usage on the terminal in that way ? Much appreciated :)

530 points by kgwgk 2y ago

behnamoh 2y ago

[flagged]

halyconWays 2y ago
[flagged]
dang 2y ago
Please make your substantive points without putting down others, and please omit such complaints about downvotes. The latter are inevitably unsubstantiated (since vote data isn't public), usually imaginary, and often provocative. If you're worried about abuse, the thing to do is email hn@ycombinator.com so we can take a look. All this is in HN's guidelines: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html. If you'd please review those and stick to them, we'd appreciate it.
- halyconWays 2y ago
  Why do you consider it "putting down others" to say that they're wrappers of other people's work, which they are?
ninth_ant 2y ago
The list of things people who deserves more credit is endless, for all intents and purposes. People make kernels, drivers, file system, drivers, languages, compilers, container systems, security infrastructure, and the list goes on. It’s literally an xkcd comic. Instead of suggesting ollama folks get too much credit, perhaps simply acknowledge the people you feel are undervalued. Because by packaging together the various bits into an easy to use system with good documentation, the ollama folks have done a lot a good for me and any attention and credit they get is richly deserved.
- IAmNotACellist 2y ago
  It's not that they get too much credit (people are excited about this tech and there will be frontrunners and winners), it's that they're a wrapper for llama.cpp and give no acknowledgement or credit to the FOSS that makes their project work.

dingnuts 2y ago

Is anyone actually getting value out of these models? I wired one up to Emacs and the local models all produce a huge volume of garbage output. Occasionally I find a hosted LLM useful but I haven't found any output from the models I can run in Ollama on my gaming PC to be useful. It's all plausible-looking but incorrect. I feel like I'm taking crazy pills when I read about others' experiences. Surely I am not alone?

remexre 2y ago
I work on compilers. A friend of mine works on webapps. I've seen Cursor give him lots of useful code, but it's never been particularly useful on any of the code of mine that I've tried it on. It seems very logical to me that there'd be orders of magnitude more training data for some domains than others, and that existing models' skill is not evenly distributed cross-domain.
- dkga 2y ago
  This. Also across languages. For example, I suppose there is a lot more content in python and javascript than Apple script, for example. (And to be fair not a lot of the python suggestions I receive are actually mindblowing good)
- q0uaur 2y ago
  i'm still patiently waiting for an easy way to point a model at some documentation, and make it actually use that. My usecase is gdscript for godot games, and all the models i've tried so far use godot 2 stuff that's just not around anymore, even if you tell it to use godot 4 it gives way too much wrong output to be useful. I wish i could just point it at the latest godot docs and have it give up to date answers. but seeing as that's still not a thing, i guess it's more complicated than i expect.
  - doctoboggan 2y ago
    It's definitely a thing already. Look up "RAG" (Retrieval Augmented Generation). Most of the popular closed source companies offer RAG services via their APIs, and you can also do it with local llms using open-webui and probably many other local UIs.

jerpint 2y ago

It’s funny because I actually use vim mostly when I don’t want LLM assisted code. Sometimes it just gets in the way. If I do, I load up cursor with vim bindings.

renewiltord 2y ago
Funny. I think the most common usage I have is using it at the command line to write commands with vim set as my EDITOR so the AI completion really helps. This will help for offline support (on planes and such).
- qup 2y ago
  Can you more specifically talk about how you use this, like with a small example?
  - renewiltord 2y ago
    Yes, I mentioned it here first and haven't changed it since (Twitter link and links include video of use) https://news.ycombinator.com/item?id=34769611 https://news.ycombinator.com/item?id=34769611 Which leads (used to lead?) here https://wiki.roshangeorge.dev/index.php/AI_Completion_In_The_Shell https://wiki.roshangeorge.dev/index.php/AI_Completion_In_The...
  - VMG 2y ago
    - composing a commit message - anything bash script related
rapind 2y ago
What I'd really like is a non-intrusive LLM running that I can put on a third monitor which sees what I'm working on, while I work, and tries to mirror and make suggestions WITHOUT interjecting itself into my editor. Then I just want a mappable prompt command (like <leader>P) where I can provide feedback / suggestions (or maybe I just do that via mic). This way I can choose to just ignore it for the most part, but if I see something interesting, I can refine it, prompt, cherry pick, etc. I really don't want autocomplete for anything.

ggerganov 2y ago

Hi HN, happy to see this here! I highly recommend to take a look at the technical details of the server implementation that enables large context usage with this plugin - I think it is interesting and has some cool ideas [0]. Also, the same plugin is available for VS Code [1]. Let me know if you have any questions about the plugin - happy to explain. Btw, the performance has improved compared to what is seen in the README videos thanks to client-side caching. [0] - https://github.com/ggerganov/llama.cpp/pull/9787 https://github.com/ggerganov/llama.cpp/pull/9787 [1] - https://github.com/ggml-org/llama.vscode https://github.com/ggml-org/llama.vscode

jerpint 2y ago
Thank you for all of your incredible contributions!
amrrs 2y ago
For those who don't know, He is the gg of `gguf`. Thank you for all your contributions! Literally the core of Ollama, LMStudio, Jan and multiple other apps!
- sergiotapia 2y ago
  well hot damn! killing it!
- halyconWays 2y ago
  [flagged]
  - madeforhnyo 2y ago
    Someone did? Could you pls share a link?
  - kamranjon 2y ago
    They collaborate together! Her name is Justine Tunney - she took her “execute everywhere” work with Cosmopolitan to make Llamafile using the llama.cpp work that Giorgi has done.

2099miles 2y ago

[flagged]

shawnlower2 2y ago
Uganda's about to get $10 billion dollars!

msoloviev 2y ago

I wonder how the "ring context" works under the hood. I have previously had (and recently messed around with again) a somewhat similar project designed for a more toy/exploratory setting (https://github.com/blackhole89/autopen https://github.com/blackhole89/autopen - demo video at https://www.youtube.com/watch?v=1O1T2q2t7i4 https://www.youtube.com/watch?v=1O1T2q2t7i4), and one of the main problems to address definitively is the question of how to manage your KV cache cleverly so you don't have to constantly perform too much expensive recomputation whenever the buffer undergoes local changes. The solution I came up with involved maintaining a tree of tokens branching whenever an alternative next token was explored, with full LLM state snapshots at fixed depth intervals so that the buffer would only have to be "replayed" for a few tokens when something changed. I wonder if there are some mathematical properties of how the important parts of the state (really, the KV cache, which can be thought of as a partial precomputation of the operation that one LLM iteration performs on the context) work that could have made this more efficient, like to avoid saving full snapshots or perhaps to be able to prune the "oldest" tokens out of a state efficiently. (edit: Georgi's comment that beat me by 3 minutes appears to be pointing at information that would go some way to answer my questions!)

deleted 2y ago

[deleted]

entelechy0 2y ago

I use this on-and-off again. It is nice that I can flip between this and Copilot by commenting out one line in my init.lua

eigenvalue 2y ago

This guy is a national treasure and has contributed so much value to the open source AI ecosystem. I hope he’s able to attract enough funding to continue making software like this and releasing it as true “no strings attached” open source.

frankfrank13 2y ago
Hard agree. This alone replaces GH Copilot/Cursor ($10+ a month)
nacs 2y ago
> This guy is a national treasure Agreed but he's an international treasure (his Github profile states Bulgaria).
feznyng 2y ago
They have: https://ggml.ai/ https://ggml.ai/ under the Company heading.
- cosmojg 2y ago
  Georgi Gerganov is the "gg" in "ggml"
  - acters 2y ago
    Also the gg in gguf

frankfrank13 2y ago

Is this more or less the same as your VSCode version? (https://github.com/ggml-org/llama.vscode https://github.com/ggml-org/llama.vscode)

mohsen1 2y ago

Terminal coding FTW! And when you're really stuck you can use DeepSeek R1 for a deeper analysis in your terminal using `askds` https://github.com/bodo-run/askds https://github.com/bodo-run/askds

opk 2y ago

Has anyone actually got this llama stuff to be usable on even moderate hardware? I find it just crashes because it doesn't find enough RAM. I've got 2G of VRAM on an AMD graphics card and 16G of system RAM and that doesn't seem to be enough. The impression I got from reading up was that it worked for most Apple stuff because the memory is unified and other than that, you need very expensive Nvidia GPUs with lots of VRAM. Are there any affordable options?

basilgohar 2y ago
I can run 7B models with Q4 quantization on a 7000 series AMD APU without GPU acceleration quite acceptably fast. This is with DDR5600 RAM which is the current roadblock for performance. Larger models work but slow down. I do have 64GB of RAM but I think 32 could work. 16GB is pushing is, but should be possible if you don't have anything else open. Memory requirements depend on numerous factors. 2GB VRAM is not enough for most GenAI stuff today.
zamadatix 2y ago
2G is pretty low and the sizes things you can get to run fast on that set up probably aren't particularly attractive. "moderate hardware" varies but you can grab a 12 GB RTX 3060 on ebay for ~$200. You can get a lot more RAM for $200 but it'll be so slow compare the the GPU I'm not sure I'd recommend it if you actually want to use things like this interactively. If "moderate hardware" is your average office PC then it's unlikely to be very usable. Anyone with a gaming GPU from the last several years should be workable though.
- horsawlarway 2y ago
  I'll second this, actually - $250 for a 12gb rtx 3060 is probably a better buy than $400 for 2xp40s for 16gb. It'd been a minute since I checked refurb prices and $250 for the rtx 3060 12gb is a good price. Easier on the rest of the system than a 2x card setup, and is probably a drop in replacement.
bhelkey 2y ago
Have you tried Ollama [1]? You should be able to run a 8b model in RAM and a 1b model in VRAM. [1] https://news.ycombinator.com/item?id=42069453 https://news.ycombinator.com/item?id=42069453

colordrops 2y ago

I've seen several posts and projects like this. Is there a summary/comparison somewhere of the various ways of running local completion/copilot?

h14h 2y ago

A little bit of a tangent, but I'm really curious what benefits could come from integrating these LLM tools more closely with data from LSPs, compilers, and other static analysis tools. Intuitively, it seems like you could provide much more context and better output as a result. Even better would be if you could fine-tune LLMs on a per-language basis and ship them alongside typical editor tooling. A problem I see w/ these AI tools is that they work much better with old, popular languages, and I worry that this will grow as a significant factor when choosing a language. Anecdotally, I see far better results when using TypeScript than Gleam, for example. It would be very cool to be able to install a Gleam-specific model that could be fed data from the LSP and compiler, and wouldn't constantly hallucinate invalid syntax. I also wonder if, with additional context & fine-tuning, you could make these models smaller and more feasible to run locally on modest hardware.

sdesol 2y ago
> work much better with old, popular languages I think this will improve significantly over time when when hardware becomes cheaper. As long as newer languages can map to older languages (syntax/function wise), we should be able generate enough synthetic data to make working with less known languages easier.
rabiescow 2y ago
you are free to contribute your own gleam llms. They're only as good as their inputs, thus if there's very few publicly available packages for a certain language that's all the input they got...

amelius 2y ago

This looks very interesting. Can this be trained on the user's codebase, or is the idea that everything must fit inside the context buffer?

larodi 2y ago
Models don’t get trained on your codebase in scenarios like Continue and Copilot assisted completion. It’s RAG at best and it depends on how the RAG is built. So all depends on how it fits in context and it will for the foreseeable time be like it as training is absolutely expensive compared to RAG.

binary132 2y ago

I am curious to see what will be possible with consumer grade hardware and more improvements to quantization over the next decade. Right now, even a 24GB gpu with the best models isn’t able to match the barely acceptable performance of hosted services I’m not willing to even pay $20 a month for.

morcus 2y ago

Looking for advice from someone who knows about the space - Suppose I'm willing to go out and buy a card for this purpose, what's a modestly priced graphics card with which I can get somewhat usable results running local LLM?

estreeper 2y ago
This is a tough question to answer, because it depends a lot on what you want to do! One way to approach it may be to look at what models you want to run and check the amount of VRAM they need. A back-of-the-napkin method taken from here[0] is: VRAM (GB) = 1.2 * number of parameters (in billions) * bits per parameter / 8 The 1.2 is just an estimation factor to account for the VRAM needed for things that aren't model parameters. Because quantization is often nearly free in terms of output quality, you should usually look for quantized versions. For example, Llama 3.2 uses 16-bit parameters but has a 4-bit quantized version, and looking at the formula above you can see that will allow you to run a 4x larger model. Having enough VRAM will allow you to run a model, but performance is dependent on a lot of other factors. For a much deeper dive into how all of this works along with price/dollar recommendations (though from last year!), Tim Dettmers wrote this excellent article: https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/ https://timdettmers.com/2023/01/30/which-gpu-for-deep-learni... Worth mentioning for the benefit of those who don't want to buy a GPU: there are also models which have been converted to run on CPU. [0] https://blog.runpod.io/understanding-vram-and-how-much-your-llm-needs/ https://blog.runpod.io/understanding-vram-and-how-much-your-...
- morcus 2y ago
  Thank you for the background!
loudmax 2y ago
The bottleneck for running LLMs on consumer grade equipment is the amount of VRAM your GPU has. VRAM is RAM that's physically built into the unit and it has much higher memory bandwidth than regular system RAM. Obviously, newer GPUs will run faster than older GPUs, but you need more VRAM to be able to run larger models. A small LLM that fits into an RTX 4060's 8GB of VRAM will run faster there than it would on an older RTX 3090. But the 3090 has 24GB of VRAM, so it can run larger LLMs that the 4060 simply can't handle. Llama.cpp can split your LLM onto multiple GPUs, or split part of it onto the CPU using system RAM, though that last option is much much slower. The more of the model you can fit into VRAM, the better. The Apple M-series macbooks have unified memory, so the GPU has higher bandwidth access to system RAM than would be available over a PCIe card. They're not as powerful as Nvidia GPUs, but they're a reasonable option for running larger LLMs. It's also worth considering AMD and Intel GPUs, but most of the development in the ML space is happening on Nvidia's CUDA architecture, so bleeding edge stuff tends to be Nvidia first and other architectures later, if at all.

awwaiid 2y ago

The blinking cursor in demo videos is giving me heart palpitations! But this is super cool. It makes me wonder how Linux is doing on M* hardware.

amelius 2y ago
> It makes me wonder how Linux is doing on M* hardware. Not so great. Zero legal status. No blessing by Apple. Can be taken down whenever Apple execs decide to.

cfiggers 2y ago

Do people with "Copilot+ PCs" get benefits running stuff like this from the much-vaunted AI coprocessors in for e.g. Snapdragon X Elite chips?

estreeper 2y ago

Very exciting - I'm a long-time vim user but most of my coworkers use VSCode, and I've been wanting to try out in-editor completion tools like this. After using it for a couple hours (on Elixir code) with Qwen2.5-Coder-3B and no attempts to customize it, this checks a lot of boxes for me: - I pretty much want fancy autocomplete: filling in obvious things and saving my fingers the work, and these suggestions are pretty good - the default keybindings work for me, I like that I can keep current line or multi-line suggestions - no concerns around sending code off to a third-party - works offline when I'm traveling - it's fast! So I don't need to remember how to run the server, I'll probably set up a script to check if it's running and if not start it up in the background and run vim, and alias vim to use that. I looked in the help documents but didn't see a way to disable the "stats" text after the suggestions, though I'm not sure it will bother me that much.

ggerganov 2y ago
Appreciate the feedback! Currently, there isn't a user-friendly way to disable the stats from showing apart from modifying the "'show_info': 0" value directly in the plugin implementation. These things will be improved with time and will become more user-friendly. A few extra optimizations will soon land which will further improve the experience: - Speculative FIM - Multiple suggestions
- tomnipotent 2y ago
  First extension I've used that perfectly autocompletes Go method receivers. First tab completes just "func (t *Type)" so then I can type the first few characters of something I'm specifically looking for or wait for the first recommendation to kick in. I hope this isn't just a coincidence from the combination of model and settings...
douglee650 2y ago
So I assume you have tried vscode vim mode. Would love to hear your thoughts. Are you on Mac/Linux or Windows?

mrinterweb 2y ago

Been using this for a couple hours, and this is really nice. It is a great alternative to something like Github Copilot. Appreciate how simple and fast this is.

mijoharas 2y ago

Can anyone compare this to Tabbyml?[0] I just set that up yesterday for emacs to check it out. The context gathering seems very interesting[1], and very vim-integrated, so I'm guessing there isn't anything very similar for Tabby. I skimmed the docs and saw some stuff about context for the Tabby chat feature[2] which I'm not super interested in using even if the docs adding sounds nice, but nothing obvious for the auto completion[3]. Does anyone have more insight or info to compare the two? As a note, I quite like that the LLM context here "follows" what you're doing. It seems like a nice idea. Does anyone know if anyone else does something similar? [0] https://www.tabbyml.com/ https://www.tabbyml.com/ [1] https://github.com/ggerganov/llama.cpp/pull/9787#issue-2572915687 https://github.com/ggerganov/llama.cpp/pull/9787#issue-25729... "global context onwards" [2] https://tabby.tabbyml.com/docs/administration/context/ https://tabby.tabbyml.com/docs/administration/context/ [3] https://tabby.tabbyml.com/docs/administration/code-completion/ https://tabby.tabbyml.com/docs/administration/code-completio...

ghthor 2y ago
I’ve been using Tabby happily since May 2023
- mijoharas 2y ago
  Have you setup the context providers, and do you find that important for getting good completions?
mijoharas 2y ago
Ahhh, it seems that tabby does use RAG and context providers for the code completion. Interesting: > During LLM inference, this context is utilized for code completion Hmmm... I wonder what's better. As I'm coding I jump and search to relevant parts of the codebase to build up my own context for solving the problem, and I expect that's likely better than RAG. Llama.vim seems to follow this model, while tabby could theoretically get at things I'm not looking at/haven't looked at recently...

s-skl 2y ago

Really awesome work! Do anyone know what's the tool/terminal configuration he's using on the video demo to embed CPU/GPU usage on the terminal in that way ? Much appreciated :)