3 ms·

Is anyone actually getting value out of these models? I wired one up to Emacs and the local models all produce a huge volume of garbage output. Occasionally I

Is anyone actually getting value out of these models? I wired one up to Emacs and the local models all produce a huge volume of garbage output. Occasionally I find a hosted LLM useful but I haven't found any output from the models I can run in Ollama on my gaming PC to be useful. It's all plausible-looking but incorrect. I feel like I'm taking crazy pills when I read about others' experiences. Surely I am not alone?

remexre 2y ago
I work on compilers. A friend of mine works on webapps. I've seen Cursor give him lots of useful code, but it's never been particularly useful on any of the code of mine that I've tried it on. It seems very logical to me that there'd be orders of magnitude more training data for some domains than others, and that existing models' skill is not evenly distributed cross-domain.
- dkga 2y ago
  This. Also across languages. For example, I suppose there is a lot more content in python and javascript than Apple script, for example. (And to be fair not a lot of the python suggestions I receive are actually mindblowing good)
- q0uaur 2y ago
  i'm still patiently waiting for an easy way to point a model at some documentation, and make it actually use that. My usecase is gdscript for godot games, and all the models i've tried so far use godot 2 stuff that's just not around anymore, even if you tell it to use godot 4 it gives way too much wrong output to be useful. I wish i could just point it at the latest godot docs and have it give up to date answers. but seeing as that's still not a thing, i guess it's more complicated than i expect.
  - doctoboggan 2y ago
    It's definitely a thing already. Look up "RAG" (Retrieval Augmented Generation). Most of the popular closed source companies offer RAG services via their APIs, and you can also do it with local llms using open-webui and probably many other local UIs.
  - mohsen1 2y ago
    Cursor can follow links
  - psytrx 2y ago
    There's llms.txt [0], but it's not gaining much popularity. My web framework of choice provides these [1], but they're not easily injected into the LLM context without much fuss. It would be a game changer if more LLM tools implemented them. [0] https://llmstxt.org/ https://llmstxt.org/ [1] https://svelte.dev/docs/llms https://svelte.dev/docs/llms
codingdave 2y ago
Yep - I don't get a ton of value out of autocompletions, but I get decent value from asking an LLM how they would approach more complex functions or features. I rarely get code back that I can copy/paste, but reading their output is something I can react to - whether it is good or bad, just having a starting point speeds up the design of new features vs. me burning time creating my first/worst draft. And that is the goal here, isn't it? To get some productivity gains? So maybe it is just a difference in perspective? Even incorrect code and bad ideas can still be helpful. It is only useless if you expect them to hand you working code.
coder543 2y ago
> I wired one up “One”? Wired up how? There is a huge difference between the best and worst. They aren’t fungible. Which one? How long ago? Did it even support FIM (fill in middle), or was it blindly guessing from the left side? Did the plugin even gather appropriate context from related files, or was it only looking at the current file? If you try Copilot or Cursor today, you can experience what “the best” looks like, which gives you a benchmark to measure smaller, dumber models and plugins against. No, Copilot and Cursor are not available for emacs, as far as I know… but if you want to understand if a technology is useful, you don’t start with the worst version and judge from that. (Not saying emacs itself is the worst… just that without more context, my assumption is that whatever plugin you probably encountered was probably using a bottom tier model, and I doubt the plugin itself was helping that model do its best.) There are some local code completion models that I think are perfectly fine, but I don’t know where you will draw the line on how good is good enough. If you can prove to yourself that the best models are good enough, then you can try out different local models and see if one of those works for you.
- yoyohello13 2y ago
  https://github.com/copilot-emacs/copilot.el https://github.com/copilot-emacs/copilot.el
- Lanedo 2y ago
  There is https://github.com/copilot-emacs/copilot.el https://github.com/copilot-emacs/copilot.el that gets copilot to work in emacs via JS glue code and binaries provided by copilot.vim. I hacked up a slim alternative localpilot.js layer that uses llama-server instead of the copilot API, so copilot.el can be used with local LLMs, but I find the copilot.el overlays kinda buggy... It'd probably be better to instead write a llamapilot.el for local LLMs from scratch for emacs.
- b5n 2y ago
  Emacs has had multiple llm integration packages available for quite awhile (relative to the rise of llms). `gptel` supports multiple providers including anthropic, openai, ollama, etc. https://github.com/karthink/gptel https://github.com/karthink/gptel
fovc 2y ago
> I feel like I'm taking crazy pills when I read about others' experiences. Surely I am not alone? You're not alone :-) I asked a very similar question about a month ago: https://news.ycombinator.com/item?id=42552653 https://news.ycombinator.com/item?id=42552653 and have continued researching since. My takeaway was that autocomplete, boiler plate, and one-off scripts are the main use cases. To use an analogy, I think the code assistants are more like an upgrade from handsaw to power tools and less like hiring a carpenter. (Which is not what the hype engine will claim). For me, only the one-off script (write-only code) use-case is useful. I've had the best results on this with Claude. Emacs abbrevs/snippets (+ choice of language) virtually eliminate the boiler plate problem, so I don't have a use for assistants there. For autocomplete, I find that LSP completion engines provide 95% of the value for 1% of the latency. Physically typing the code is a small % of my time/energy, so the value is more about getting the right names, argument order, and other fiddly details I may not remember exactly. But I find, that LSP-powered autocomplete and tooltips largely solve those challenges.
- barrell 2y ago
  I think you make a very good point about your existing devenv. I recently turned off GitHub copilot after maybe 2 years of use — I didn’t realize how often I was using its completions over LSPs. Quality of Life went up massively. LSPs and nvim-cmp have come a long way (although one of these days I’ll try blink.cmp)
- sdesol 2y ago
  > like an upgrade from handsaw to power tools and less like hiring a carpenter. (Which is not what the hype engine will claim). I 100% agree with the not hiring a carpenter part but we need a better way to describe the improvement over just a handsaw. If you have domain knowledge, it can become an incredible design aid/partner. Here is a real world example as to how it is changing things for me. I have a TreeTable component which I built 100% with LLM and when I need to update it, I just follow the instructions in this chat: http://beta.gitsense.com/?chat=dd997ccd-5b37-4591-9200-b975f274450c http://beta.gitsense.com/?chat=dd997ccd-5b37-4591-9200-b975f... Right now, I am thinking about adding folders to organize chats, and here is the chat with DeepSeek for that feature: http://beta.gitsense.com/?chat=3a94ce40-86f2-4e68-b5d7-88d337bd11ba http://beta.gitsense.com/?chat=3a94ce40-86f2-4e68-b5d7-88d33... I'm thoroughly impressed as it suggested data structures and more for me to think about. And here I am asking it to review what was discussed to make the information easier to understand. http://beta.gitsense.com/?chat=8c6bf5db-49a7-4511-990c-5e6ad3a41955 http://beta.gitsense.com/?chat=8c6bf5db-49a7-4511-990c-5e6ad... All of this cost me less than a penny. I'm still waiting for my Anthropic API limit to reset and I'm going to ask Sonnet for feedback as well, and I figure that will cost me 5 cents. I fully understand the not hiring a carpenter part, but I think what LLMs bring to the table is SO MUCH more than an upgrade to a power tool. If you know what you need and can clearly articulate it well enough, there really is no limit to what you can build with proper instructions, provided the solution is in its training data and you have a good enough BS detector.
tomr75 2y ago
try cursor
sangnoir 2y ago
> Is anyone actually getting value out of these models? I've found incredible value in having LLMs help me write unit tests. The quality of the test code is far from perfect, but AI tooling - Claude Sonnet specifically - is good at coming up with reasonable unit test cases after I've written the code under test (sue me, TDD zealots). I probably have to fix 30% of the tests and expand the test cases, but I'd say it cuts the number if test code lines I author by more than 80%. This has decreased the friction so much, I've added Continuous Integration to small, years-old personal projects that had no tests before. I've found lesser value with refactoring and adding code docs, but that's more of autocomplete++ using natural language rather than AST-derived code.
righthand 2y ago
Honestly just disabled my TabNine plugin and have found that LSP server is good enough for 99% of what I do. I really don’t need hypothetical output suggested to me. I’m comfortable reading docs though so others may feel different.
whimsicalism 2y ago
i don't find value from the models that it makes economical sense to self-host. i do get value out of a llama 70b, for instance, though
colonial 2y ago
I do not, but I suspect that's mainly because I far and away prefer statically typed languages with powerful LSP implementations. Copilot, Ollama, and the others have all been strictly inferior to rust-analyzer. The suggested code is often just straight up invalid and takes just long enough to be annoying. Compare that to just typing '.'/'::' + a few characters to fuzzy-select what I'm looking for + enter. ETA: Both did save me a few seconds here and there when obvious "repetition with a few tweaks each line" was involved, but to me that is not worth a monthly subscription or however much wall power my GPU was consuming.

remexre 2y ago

I work on compilers. A friend of mine works on webapps. I've seen Cursor give him lots of useful code, but it's never been particularly useful on any of the code of mine that I've tried it on. It seems very logical to me that there'd be orders of magnitude more training data for some domains than others, and that existing models' skill is not evenly distributed cross-domain.

dkga 2y ago
This. Also across languages. For example, I suppose there is a lot more content in python and javascript than Apple script, for example. (And to be fair not a lot of the python suggestions I receive are actually mindblowing good)
q0uaur 2y ago
i'm still patiently waiting for an easy way to point a model at some documentation, and make it actually use that. My usecase is gdscript for godot games, and all the models i've tried so far use godot 2 stuff that's just not around anymore, even if you tell it to use godot 4 it gives way too much wrong output to be useful. I wish i could just point it at the latest godot docs and have it give up to date answers. but seeing as that's still not a thing, i guess it's more complicated than i expect.
- doctoboggan 2y ago
  It's definitely a thing already. Look up "RAG" (Retrieval Augmented Generation). Most of the popular closed source companies offer RAG services via their APIs, and you can also do it with local llms using open-webui and probably many other local UIs.
- mohsen1 2y ago
  Cursor can follow links
- psytrx 2y ago
  There's llms.txt [0], but it's not gaining much popularity. My web framework of choice provides these [1], but they're not easily injected into the LLM context without much fuss. It would be a game changer if more LLM tools implemented them. [0] https://llmstxt.org/ https://llmstxt.org/ [1] https://svelte.dev/docs/llms https://svelte.dev/docs/llms

codingdave 2y ago

Yep - I don't get a ton of value out of autocompletions, but I get decent value from asking an LLM how they would approach more complex functions or features. I rarely get code back that I can copy/paste, but reading their output is something I can react to - whether it is good or bad, just having a starting point speeds up the design of new features vs. me burning time creating my first/worst draft. And that is the goal here, isn't it? To get some productivity gains? So maybe it is just a difference in perspective? Even incorrect code and bad ideas can still be helpful. It is only useless if you expect them to hand you working code.

coder543 2y ago

> I wired one up “One”? Wired up how? There is a huge difference between the best and worst. They aren’t fungible. Which one? How long ago? Did it even support FIM (fill in middle), or was it blindly guessing from the left side? Did the plugin even gather appropriate context from related files, or was it only looking at the current file? If you try Copilot or Cursor today, you can experience what “the best” looks like, which gives you a benchmark to measure smaller, dumber models and plugins against. No, Copilot and Cursor are not available for emacs, as far as I know… but if you want to understand if a technology is useful, you don’t start with the worst version and judge from that. (Not saying emacs itself is the worst… just that without more context, my assumption is that whatever plugin you probably encountered was probably using a bottom tier model, and I doubt the plugin itself was helping that model do its best.) There are some local code completion models that I think are perfectly fine, but I don’t know where you will draw the line on how good is good enough. If you can prove to yourself that the best models are good enough, then you can try out different local models and see if one of those works for you.

yoyohello13 2y ago
https://github.com/copilot-emacs/copilot.el https://github.com/copilot-emacs/copilot.el
Lanedo 2y ago
There is https://github.com/copilot-emacs/copilot.el https://github.com/copilot-emacs/copilot.el that gets copilot to work in emacs via JS glue code and binaries provided by copilot.vim. I hacked up a slim alternative localpilot.js layer that uses llama-server instead of the copilot API, so copilot.el can be used with local LLMs, but I find the copilot.el overlays kinda buggy... It'd probably be better to instead write a llamapilot.el for local LLMs from scratch for emacs.
b5n 2y ago
Emacs has had multiple llm integration packages available for quite awhile (relative to the rise of llms). `gptel` supports multiple providers including anthropic, openai, ollama, etc. https://github.com/karthink/gptel https://github.com/karthink/gptel

fovc 2y ago

> I feel like I'm taking crazy pills when I read about others' experiences. Surely I am not alone? You're not alone :-) I asked a very similar question about a month ago: https://news.ycombinator.com/item?id=42552653 https://news.ycombinator.com/item?id=42552653 and have continued researching since. My takeaway was that autocomplete, boiler plate, and one-off scripts are the main use cases. To use an analogy, I think the code assistants are more like an upgrade from handsaw to power tools and less like hiring a carpenter. (Which is not what the hype engine will claim). For me, only the one-off script (write-only code) use-case is useful. I've had the best results on this with Claude. Emacs abbrevs/snippets (+ choice of language) virtually eliminate the boiler plate problem, so I don't have a use for assistants there. For autocomplete, I find that LSP completion engines provide 95% of the value for 1% of the latency. Physically typing the code is a small % of my time/energy, so the value is more about getting the right names, argument order, and other fiddly details I may not remember exactly. But I find, that LSP-powered autocomplete and tooltips largely solve those challenges.

barrell 2y ago
I think you make a very good point about your existing devenv. I recently turned off GitHub copilot after maybe 2 years of use — I didn’t realize how often I was using its completions over LSPs. Quality of Life went up massively. LSPs and nvim-cmp have come a long way (although one of these days I’ll try blink.cmp)
sdesol 2y ago
> like an upgrade from handsaw to power tools and less like hiring a carpenter. (Which is not what the hype engine will claim). I 100% agree with the not hiring a carpenter part but we need a better way to describe the improvement over just a handsaw. If you have domain knowledge, it can become an incredible design aid/partner. Here is a real world example as to how it is changing things for me. I have a TreeTable component which I built 100% with LLM and when I need to update it, I just follow the instructions in this chat: http://beta.gitsense.com/?chat=dd997ccd-5b37-4591-9200-b975f274450c http://beta.gitsense.com/?chat=dd997ccd-5b37-4591-9200-b975f... Right now, I am thinking about adding folders to organize chats, and here is the chat with DeepSeek for that feature: http://beta.gitsense.com/?chat=3a94ce40-86f2-4e68-b5d7-88d337bd11ba http://beta.gitsense.com/?chat=3a94ce40-86f2-4e68-b5d7-88d33... I'm thoroughly impressed as it suggested data structures and more for me to think about. And here I am asking it to review what was discussed to make the information easier to understand. http://beta.gitsense.com/?chat=8c6bf5db-49a7-4511-990c-5e6ad3a41955 http://beta.gitsense.com/?chat=8c6bf5db-49a7-4511-990c-5e6ad... All of this cost me less than a penny. I'm still waiting for my Anthropic API limit to reset and I'm going to ask Sonnet for feedback as well, and I figure that will cost me 5 cents. I fully understand the not hiring a carpenter part, but I think what LLMs bring to the table is SO MUCH more than an upgrade to a power tool. If you know what you need and can clearly articulate it well enough, there really is no limit to what you can build with proper instructions, provided the solution is in its training data and you have a good enough BS detector.

tomr75 2y ago

try cursor

sangnoir 2y ago

> Is anyone actually getting value out of these models? I've found incredible value in having LLMs help me write unit tests. The quality of the test code is far from perfect, but AI tooling - Claude Sonnet specifically - is good at coming up with reasonable unit test cases after I've written the code under test (sue me, TDD zealots). I probably have to fix 30% of the tests and expand the test cases, but I'd say it cuts the number if test code lines I author by more than 80%. This has decreased the friction so much, I've added Continuous Integration to small, years-old personal projects that had no tests before. I've found lesser value with refactoring and adding code docs, but that's more of autocomplete++ using natural language rather than AST-derived code.

righthand 2y ago

Honestly just disabled my TabNine plugin and have found that LSP server is good enough for 99% of what I do. I really don’t need hypothetical output suggested to me. I’m comfortable reading docs though so others may feel different.

whimsicalism 2y ago

i don't find value from the models that it makes economical sense to self-host. i do get value out of a llama 70b, for instance, though

colonial 2y ago

I do not, but I suspect that's mainly because I far and away prefer statically typed languages with powerful LSP implementations. Copilot, Ollama, and the others have all been strictly inferior to rust-analyzer. The suggested code is often just straight up invalid and takes just long enough to be annoying. Compare that to just typing '.'/'::' + a few characters to fuzzy-select what I'm looking for + enter. ETA: Both did save me a few seconds here and there when obvious "repetition with a few tweaks each line" was involved, but to me that is not worth a monthly subscription or however much wall power my GPU was consuming.