3 ms·

Any agentic dev software you could recommend that runs well with local models? I’ve been using Cursor and I’m kind of disappointed. I get better results just g

by nico 1y ago

Any agentic dev software you could recommend that runs well with local models? I’ve been using Cursor and I’m kind of disappointed. I get better results just going back and forth between the editor and ChatGPT I tried localforge and aider, but they are kinda slow with local models

ynniv 1y ago
https://github.com/block/goose https://github.com/block/goose
jabroni_salad 1y ago
Do you have any other interface for the model? what kind of tokens/sec are you getting? Try hooking aider up to gemini and see how the speed is. I have noticed that people in the localllama scene do not like to talk about their TPS.
- nico 1y ago
  The models feel pretty snappy when interacting with them directly via ollama, not sure about the TPS However I've also ran into 2 things: 1) most models don't support tools, sometimes it's hard to find a version of the model that correctly uses tools, 2) even with good TPS, since the agents are usually doing chain-of-thought and running multiple chained prompts, the experience feels slow - this is even true with Cursor using their models/apis
- segmondy 1y ago
  People have all sorts of hardware, TPS is meaningless without the full spec of the hardware, and GPU is not the only thing, CPU, ram speed, memory channel, PCIe speed, inference software, partial CPU offload? RPC? even OS, all of these things add up. So if someone tells you TPS for a given model, it's meaningless unless you understand their entire setup.
zackify 1y ago
I used devstral today with cline and open hands. Worked great in both. About 1 minute initial prompt processing time on an m4 max Using LM studio because the ollama api breaks if you set the context to 128k.
- elAhmo 1y ago
  How is it great that it takes 1 minute for initial prompt processing?
  - cheema33 1y ago
    That time is just for the very first prompt. It is basically the startup time for the model. Once it is loaded, it is much much faster in responding to your queries. Depending on your hardware of course.
  - zackify 1y ago
    Haha great as in surprisingly good at some simple things that nothing has been able to do locally for me. The 1 minute first token sucks and has me dreaming for the day of 3-4x the bandwidth
- nico 1y ago
  Have you tried using mlx or Simon Wilson’s llm? https://llm.datasette.io/en/stable/ https://llm.datasette.io/en/stable/ https://simonwillison.net/tags/llm/ https://simonwillison.net/tags/llm/
  - zackify 1y ago
    On lm studio I was using mlx
asimovDev 1y ago
you can use ollama in VS Code's copilot. I haven't personally tried it but I am interested in how it would perform with devstral
mrshu 1y ago
ra-aid works pretty well with Ollama (haven't tried it with Devstral yet though) https://docs.ra-aid.ai/configuration/ollama/ https://docs.ra-aid.ai/configuration/ollama/
ivanvanderbyl 1y ago
I’ve been playing around with Zed, supports local and cloud models, really fast, nice UX. It does lack some of the deeper features of VSCode/Cursor but very capable.

by nico 1y ago

ynniv 1y ago

https://github.com/block/goose https://github.com/block/goose

jabroni_salad 1y ago

Do you have any other interface for the model? what kind of tokens/sec are you getting? Try hooking aider up to gemini and see how the speed is. I have noticed that people in the localllama scene do not like to talk about their TPS.

nico 1y ago
The models feel pretty snappy when interacting with them directly via ollama, not sure about the TPS However I've also ran into 2 things: 1) most models don't support tools, sometimes it's hard to find a version of the model that correctly uses tools, 2) even with good TPS, since the agents are usually doing chain-of-thought and running multiple chained prompts, the experience feels slow - this is even true with Cursor using their models/apis
segmondy 1y ago
People have all sorts of hardware, TPS is meaningless without the full spec of the hardware, and GPU is not the only thing, CPU, ram speed, memory channel, PCIe speed, inference software, partial CPU offload? RPC? even OS, all of these things add up. So if someone tells you TPS for a given model, it's meaningless unless you understand their entire setup.

zackify 1y ago

I used devstral today with cline and open hands. Worked great in both. About 1 minute initial prompt processing time on an m4 max Using LM studio because the ollama api breaks if you set the context to 128k.

elAhmo 1y ago
How is it great that it takes 1 minute for initial prompt processing?
- cheema33 1y ago
  That time is just for the very first prompt. It is basically the startup time for the model. Once it is loaded, it is much much faster in responding to your queries. Depending on your hardware of course.
- zackify 1y ago
  Haha great as in surprisingly good at some simple things that nothing has been able to do locally for me. The 1 minute first token sucks and has me dreaming for the day of 3-4x the bandwidth
nico 1y ago
Have you tried using mlx or Simon Wilson’s llm? https://llm.datasette.io/en/stable/ https://llm.datasette.io/en/stable/ https://simonwillison.net/tags/llm/ https://simonwillison.net/tags/llm/
- zackify 1y ago
  On lm studio I was using mlx

asimovDev 1y ago

you can use ollama in VS Code's copilot. I haven't personally tried it but I am interested in how it would perform with devstral

mrshu 1y ago

ra-aid works pretty well with Ollama (haven't tried it with Devstral yet though) https://docs.ra-aid.ai/configuration/ollama/ https://docs.ra-aid.ai/configuration/ollama/

ivanvanderbyl 1y ago

I’ve been playing around with Zed, supports local and cloud models, really fast, nice UX. It does lack some of the deeper features of VSCode/Cursor but very capable.