5 ms·

If anyone is looking for a free/local alternative Continue + Ollama is acceptable. If you're just doing run of the mill programming it will work well out of the

If anyone is looking for a free/local alternative Continue + Ollama is acceptable. If you're just doing run of the mill programming it will work well out of the box. I'm glad it's open source so I was able to fix most of the issues I had with it and now my copy is in a great place. The documentation is in places many versions behind the actual code so it can be tough to figure out how to set things up when you're venturing off the beaten path. That all being said the granularity of control you have when using local models leads to an experience that's far better than Cursor/Copilot, I really enjoy that it reads my mind a lot of the time now (because I have prompt engineered it to know how I think). Ultimately, isn't this just the way of things? https://thot-experiment.github.io/forever-problems/?set%20up%20my%20dev%20environment https://thot-experiment.github.io/forever-problems/?set%20up...

TacticalCoder 2y ago
On what kind of hardware/GPU are you running that locally?
- deathtrader666 2y ago
  M1 MacBook Pro is sufficient for a Qwen model.
  - indigodaddy 2y ago
    with how much ram though?
    - int_19h 2y ago
      For Qwen-32B, 32Gb should be quite sufficient to run it at 4-bit quantization with full context.
      - indigodaddy 2y ago
        Thanks, I need to figure out something to run on M4 Mac Mini standard 16G
        int_19h 2y ago
        You should be looking at 7-8b sized models, then. https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct is considered pretty strong for its size. That said, you shouldn't expect more than glorified autocompletion at that point.
        thot_experiment 2y ago
        I'm not familiar with the practical memory requirements on Macs but I suspect that with 16gb of integrated ram you won't have issues running 14B models even at q6_k, certainly would be fine at q4 and it's definitely going to be capable of writing code based on instruction as well as minor refactoring, generating docstrings etc.
        int_19h 2y ago
        The model itself will fit just fine, of course, but you'll also want a large context for coding. And then since it's integrated RAM, it's also used by everything else running on your system - like, say, your IDE, your compiler etc, which are all fairly memory hungry. Also keep in mind that, even though it's "unified memory", the OS enforces a certain quota for the GPU. If I remember correctly, it's something like 2/3 of the overall RAM.
- thot_experiment 2y ago
  2x 1080Ti, ~25t/s
Zopieux 2y ago
Tried a few models and was extremely disappointed by how dumb the completions are. Basically useless. What local ollama-available models do you recommend?
- sorenjan 2y ago
  Have you tried Qwen 2.5 coder? I've only done very little testing, but it seemed to work pretty well. I used the 14B version. https://ollama.com/library/qwen2.5-coder https://ollama.com/library/qwen2.5-coder
meta_x_ai 2y ago
A software developer's time is much more precious than wasting time on sub-optimal models. Open Weights models has it's place (in training custom agents and custom services), but if you are knowledge worker, using a model even 5% less than SOTA is extremely dumb
- thot_experiment 2y ago
  100% disagree with this take, the flexibility in controlling the prompt leads to QwenCoder2.5-32b outperforming gpt-o1 and claude sonnet 3.5 for nearly everything that I use it for (true for Gemma-27b and llama3.3-70b, though in this context I'm almost always using the former). A specialist model that's specifically prompted to do the correct thing will outperform a SOTA generic model with a one size fits all system prompt. This is why small autocomplete models can very obviously outperform larger models at that specific task. I am speaking 100% from experience and ignoring all benchmarks in forming this view btw, so maybe it's just my specific situation. Also, in general I don't find the difference between SOTA models and local models to be that significant in the real world even when used in the exact same way.
  - k__ 2y ago
    Sounds great. Does this run with VSCode and how hard is it to set this up?
    - thot_experiment 2y ago
      yes, the vscode extension is a one click install, so is ollama which is a separate project that provides local inference you'll then have to download a model, which ollama makes very easy. choosing which one will depend on your hardware but the biggest QwenCoder2.5 you can fit is a very solid starting place. it's not ready for your grandma, but it's easy enough that I'd trust a junior dev to be able to get it done
      - k__ 2y ago
        What's the extension name?
deleted 2y ago
[deleted]
flyingpenguin 2y ago
I found the plugin did not prompt the model often enough. I would finish writing aline, and go to a new line and sit there waiting to see what the AI thought would be next, only to realize continue wasn't prompting...
scosman 2y ago
Acceptable UX maybe, but the gap between Sonnet 3.5 and open models isn’t worth it. I know people are going to pitch qwen coder 72b, but it’s still a long way off on benchmarks and my time matters more.
- thot_experiment 2y ago
  You're the second person in this thread to make this point, what are you using it for? I find the difference is basically negligible (in the sense that both get the busywork right and both fail at anything complicated)
  - scosman 2y ago
    yeah, Sonnet goes past that. 300+ line changes in 20 seconds. You have to review it, but generally it's right. It's infinitely faster than the time to look at docs and do it myself. Sure it's busywork. But it's a lot of busywork very fast.
    - dingnuts 2y ago
      I gotta find a new trade
    - thot_experiment 2y ago
      Well it's definitely not infinitely faster since you're having to review it, but we're talking about the delta between Sonnet and Qwen/Mistral/Llama or whatever, not doing it manually. I'm really curious what your problem domain is, like specifically what sort of code are you asking it to change and what changes are you asking for. I just gave o1 and Sonnet a total layup question (optimization that had a huge win simply by filtering an array before sorting it vs the other way around) and neither model got the solution right, both of them came up with ~hundred lines of code, neither model's code worked on the first try. It took me like 10 minutes to refactor and optimize the code for a 6x speedup and it would take longer than that to debug the AI code to even make it run. (I spent 10 minutes prompting/editing to try to get the generated solutions to run) Also the initial code was 11 sloc, my solution is 14 sloc, and claude was 70 sloc and o1 was 93. idfk, i just don't think we're there yet

TacticalCoder 2y ago

On what kind of hardware/GPU are you running that locally?

deathtrader666 2y ago
M1 MacBook Pro is sufficient for a Qwen model.
- indigodaddy 2y ago
  with how much ram though?
  - int_19h 2y ago
    For Qwen-32B, 32Gb should be quite sufficient to run it at 4-bit quantization with full context.
    - indigodaddy 2y ago
      Thanks, I need to figure out something to run on M4 Mac Mini standard 16G
      - int_19h 2y ago
        You should be looking at 7-8b sized models, then. https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct is considered pretty strong for its size. That said, you shouldn't expect more than glorified autocompletion at that point.
        thot_experiment 2y ago
        I'm not familiar with the practical memory requirements on Macs but I suspect that with 16gb of integrated ram you won't have issues running 14B models even at q6_k, certainly would be fine at q4 and it's definitely going to be capable of writing code based on instruction as well as minor refactoring, generating docstrings etc.
        int_19h 2y ago
        The model itself will fit just fine, of course, but you'll also want a large context for coding. And then since it's integrated RAM, it's also used by everything else running on your system - like, say, your IDE, your compiler etc, which are all fairly memory hungry. Also keep in mind that, even though it's "unified memory", the OS enforces a certain quota for the GPU. If I remember correctly, it's something like 2/3 of the overall RAM.
thot_experiment 2y ago
2x 1080Ti, ~25t/s

Zopieux 2y ago

Tried a few models and was extremely disappointed by how dumb the completions are. Basically useless. What local ollama-available models do you recommend?

sorenjan 2y ago
Have you tried Qwen 2.5 coder? I've only done very little testing, but it seemed to work pretty well. I used the 14B version. https://ollama.com/library/qwen2.5-coder https://ollama.com/library/qwen2.5-coder

meta_x_ai 2y ago

A software developer's time is much more precious than wasting time on sub-optimal models. Open Weights models has it's place (in training custom agents and custom services), but if you are knowledge worker, using a model even 5% less than SOTA is extremely dumb

thot_experiment 2y ago
100% disagree with this take, the flexibility in controlling the prompt leads to QwenCoder2.5-32b outperforming gpt-o1 and claude sonnet 3.5 for nearly everything that I use it for (true for Gemma-27b and llama3.3-70b, though in this context I'm almost always using the former). A specialist model that's specifically prompted to do the correct thing will outperform a SOTA generic model with a one size fits all system prompt. This is why small autocomplete models can very obviously outperform larger models at that specific task. I am speaking 100% from experience and ignoring all benchmarks in forming this view btw, so maybe it's just my specific situation. Also, in general I don't find the difference between SOTA models and local models to be that significant in the real world even when used in the exact same way.
- k__ 2y ago
  Sounds great. Does this run with VSCode and how hard is it to set this up?
  - thot_experiment 2y ago
    yes, the vscode extension is a one click install, so is ollama which is a separate project that provides local inference you'll then have to download a model, which ollama makes very easy. choosing which one will depend on your hardware but the biggest QwenCoder2.5 you can fit is a very solid starting place. it's not ready for your grandma, but it's easy enough that I'd trust a junior dev to be able to get it done
    - k__ 2y ago
      What's the extension name?

deleted 2y ago

[deleted]

flyingpenguin 2y ago

I found the plugin did not prompt the model often enough. I would finish writing aline, and go to a new line and sit there waiting to see what the AI thought would be next, only to realize continue wasn't prompting...

scosman 2y ago

Acceptable UX maybe, but the gap between Sonnet 3.5 and open models isn’t worth it. I know people are going to pitch qwen coder 72b, but it’s still a long way off on benchmarks and my time matters more.

thot_experiment 2y ago
You're the second person in this thread to make this point, what are you using it for? I find the difference is basically negligible (in the sense that both get the busywork right and both fail at anything complicated)
- scosman 2y ago
  yeah, Sonnet goes past that. 300+ line changes in 20 seconds. You have to review it, but generally it's right. It's infinitely faster than the time to look at docs and do it myself. Sure it's busywork. But it's a lot of busywork very fast.
  - dingnuts 2y ago
    I gotta find a new trade
  - thot_experiment 2y ago
    Well it's definitely not infinitely faster since you're having to review it, but we're talking about the delta between Sonnet and Qwen/Mistral/Llama or whatever, not doing it manually. I'm really curious what your problem domain is, like specifically what sort of code are you asking it to change and what changes are you asking for. I just gave o1 and Sonnet a total layup question (optimization that had a huge win simply by filtering an array before sorting it vs the other way around) and neither model got the solution right, both of them came up with ~hundred lines of code, neither model's code worked on the first try. It took me like 10 minutes to refactor and optimize the code for a 6x speedup and it would take longer than that to debug the AI code to even make it run. (I spent 10 minutes prompting/editing to try to get the generated solutions to run) Also the initial code was 11 sloc, my solution is 14 sloc, and claude was 70 sloc and o1 was 93. idfk, i just don't think we're there yet