5 ms·

This is basically my experience as well. I have a moderately recent but high spec desktop (Radeon 6900 XT with 16 GB VRAM, Ryzen 9 7900X 12-core, 64 GB system R

by saghm 4mo ago

This is basically my experience as well. I have a moderately recent but high spec desktop (Radeon 6900 XT with 16 GB VRAM, Ryzen 9 7900X 12-core, 64 GB system RAM), and I tried out some recommended models with ollama a month or two ago. Anything not geared specifically towards coding seemed to struggled with actually making tool calls instead of just stating the actions they would take without making them (and trying to get help from them to explain what I needed to configure to change that behavior was useless; qwen refused to believe that it was running in ollama and insisted that it was running from the Alibaba cloud without access to my local system), and the models intended for coding were barely thinking faster than I could type (if they had any ability to show thinking at all). The best "free" experience I've found is using OpenCode with Big Pickle. It's not especially smart, so it often won't produce the correct result the first time, but the free tier is generous enough that I don't think I've hit the limit more than twice over around a month with frequent multi-hour sessions. If running locally is truly the goal, it's not going to fit the bill, but if the goal is just "get the best experience without having to pay for a sub or tokens", it's the least bad option I've found so far.

alexpotato 4mo ago
> The best "free" experience I've found is using OpenCode with Big Pickle. They now offer DeepSeek V4 Flash for free and it def feels like a step up.
rapind 4mo ago
> The best "free" experience I've found is using OpenCode with Big Pickle. I have absolutely zero interest in free. I honestly don't think I'm even remotely in the same demographic as people using free tiers / models. I want to pay. I don't want my data used for training. I want it to be open. I want it to be consistently up (more than Claude!). I want it to be fast. I don't want it to be subsidized as that's just an excuse for shitty quality. Deepseek flash knocks it out of the park on all of these except you're data is used in training. I'm fine with it being hosted since there's no way I'm using it 24/7, but data MUST be private. Basically I want Hetzner and OVH to run open model clouds. I'm convinced this is going to happen eventually when everyone realizes this is a commodity.
- Bnjoroge 4mo ago
  You can specify which providers you want to serve your model in OpenRouter. Then you can chose US-based ones.
- aamoscodes 4mo ago
  You can pay, and also use deepseek-v4-flash. OpenRouter even lets you "block" or limit your usage to providers that don't train on data. Since the weights are open, other companies are already serving the model on non-DeepSeek owned hardware: https://openrouter.ai/deepseek/deepseek-v4-flash https://openrouter.ai/deepseek/deepseek-v4-flash
  - rapind 4mo ago
    Good to know. I hadn't checks since early is DS4's launch when they were the only provide (I think maybe there was one other, but they also trained on your data). I see several private options now.
  - fc417fc802 4mo ago
    > OpenRouter even lets you "block" or limit your usage to providers that don't train on data. More than that, they have various zero data retention options and provide a convenient json list of them.
    - larodi 4mo ago
      The fact OpenRouter strips https to reroute screams danger already.
      - fc417fc802 4mo ago
        What do you mean? Are you objecting that they communicate with the provider on your behalf? But how else would you design such a system? Plumbing you straight through would require nonstandard certificate juggling and they wouldn't be able to implement their core service of providing a standardized API nor could they transparently route your request to the fastest / cheapest / whatever provider on the fly nor could they implement transparent fallback nor could they implement their policy of not billing you if the response from the provider is invalid. Also the chosen provider could fingerprint your network stack if you communicated directly. The routing service is acting as a proxy and for most providers fully anonymizes requests (it does send a stable uid to some of them though).
spockz 4mo ago
For what it is worth, I’m on a similar machine. (9070XT,5900X) and found a lot of performance improvement over ollama by compiling llama.cpp and running with —no-mmap and —perf. The context is still quite small though. With online models I use contexts of at least 200k which is useful for longer running/more complicated commands. Locally I haven’t gone much further than 8k. That is sufficient for small changes on small code bases. And you need condensed tool output. I haven’t tried any tool that compresses the tokens yet.
- echelon 4mo ago
  I would rather we give up the idea of running open models on RTX cards and instead focus on running much bigger open models on H200s. 1. The hardware will eventually catch up. 2. This keeps the delta between frontier models smaller. 3. We can still fine tune and own the weights. 4. The models will be more useful, faster, and reliable. RTX is hobbyist tier, not professional tier. Gated cloud models from hyperscalers treat us like hobbyists in their own right. We need equivalent scale models, but open.
  - MrLeap 4mo ago
    There's a lot more professionals that have RTX cards than H200s. You're inevitably see more development and experimentation on things actual humans have lmao.
  - SR2Z 4mo ago
    That GPU costs 25k which means you really should have a rack to put it in. It's not realistic.
  - zozbot234 4mo ago
    H200s and other enterprise datacenter GPUs are completely overkill in any realistic single- or few-users inference scenario. They're hugely unbalanced towards compute capacity which will go almost entirely unused (i.e. wasted) unless you're running huge batches on a continued basis. I've argued many times that local inference engines should support batched inference on a somewhat smaller scale for a variety of reasons (especially given the unexpected effectiveness of SSD streamed inference with larger-than-RAM models), but even I don't think we can realistically go to 300x or so for real-time inference, which is the range that pencils out quite consistently from a simple roofline model of these datacenter cards.
ryukoposting 4mo ago
I found that, with the heavily quantized Qwen3 models I can cram onto my 3060 Ti, telling the model to use its tools in the system prompt made it a lot more likely to actually do it. YMMV of course, but give it a shot.
- saghm 4mo ago
  I did try this, and it was pretty hit-or-miss still. I even went as far as configuring context for Zed to inject into all conversations saying stuff like "If you need to read a file, call read_file NOW. Do not say you will read it", and it still didn't really make a huge difference.
redmalang 4mo ago
Try llama.cpp it seems to be a lot more performant and a lot more hackable. Also I'm surprised how substantial the impact of some of the inference configs (beyond just temp) can have, though this is much more model specific.
boppo1 4mo ago
I have almost your system specs, how do they work for non-coding stuff like chat/knowledge/discussion? I've been using models to talk through social stuff I'm anxious about but dont want to annoy my friends with and it's been amazing, but I don't want to share that info with google/openai/anthropic anymore. I shouldn't have in the first place, but I couldn't help it, the exercise was too interesting.
- fc417fc802 4mo ago
  You can test the open models for yourself using the various router services. Those also make it easy to use providers other than the major players.
markussss 4mo ago
My system is quite similar to your, my GPU is a 6950 XT and CPU a Ryzen 5 2600x, same amount of RAM, and I feel your pain. It sounds very similar to my experience from a few months ago. When it comes to tool calling, there are multiple possible issues; some models have borked templates bundled with the model file, some models are not trained on tool calling, some agent harnesses doesn't support the tool call output from the model very well, some quantizations ruin the models' abilities to call tools. My suggestions if you want to further experiment with local models are to use llama.cpp instead of ollama [1], learn a little about the parameters that tune how much VRAM is used [2], look online for jinja template fixes for the model you're testing [3], and choose a model that was designed to do the task you want to achieve, with as high quantization as you can fit. The maximum model size you can run is VRAM + RAM, although you want as little of the model to be in system RAM as possible. I'm running North Mini Code IQ3_XXS with some tuned parameters to fit my current tasks, and while it is not perfect for everything, it has not failed any tool calls I've asked it to make, or that it figured it should make on its own. [1]: https://sleepingrobots.com/dreams/stop-using-ollama/ https://sleepingrobots.com/dreams/stop-using-ollama/ [2]: https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md https://github.com/ggml-org/llama.cpp/blob/master/tools/serv... [3]: https://gist.github.com/jscott3201/e4b155885cc68c038d6ac8909a3bd9fe https://gist.github.com/jscott3201/e4b155885cc68c038d6ac8909...
- calgoo 4mo ago
  "My suggestions if you want to further experiment with local models are to use llama.cpp instead of ollama" Or at least LM Studio if you want to play around with a lot of different models. Im currently using it with my 7800xt and Vulcan as i found it left my OS more stable ROCm does. I had a few system crashes with ROCm and running out of VRAM for the OS.
- itissid 4mo ago
  Interesting. Making low latency correct tool calls correctly is pretty important in voice AI cascading models(STT LLM TTS). Realtime Models are still 2x the cost and there are only 2 providers openai and google that are in the race. For cost control it has to be cascading models For llms Sadly the only model right now that fits the bill for LLM is GPT 4.1 and it’s standard in my stack because thinking models have unacceptable latency(>=1 sec) even though they are good at tool calling. The main issue with 4.1 is that it can make still mistakes and prompt prose has to be tuned quite a bit. I wonder if any local models can be tuned to match the response time and tool calling while supporting many languages.
alfiedotwtf 4mo ago
> qwen refused to believe that it was running in ollama and insisted that it was running from the Alibaba cloud without access to my local system Sounds like you were either running at a too-low quant, or you were trying to do Agents with something like Qwen 3.5 9B? Qwen 3.6 27B at Q4_K_M I can have that at running all night after a single one-shot a, Anne when I come back in the morning, it’s done

by saghm 4mo ago

alexpotato 4mo ago

> The best "free" experience I've found is using OpenCode with Big Pickle. They now offer DeepSeek V4 Flash for free and it def feels like a step up.

rapind 4mo ago

> The best "free" experience I've found is using OpenCode with Big Pickle. I have absolutely zero interest in free. I honestly don't think I'm even remotely in the same demographic as people using free tiers / models. I want to pay. I don't want my data used for training. I want it to be open. I want it to be consistently up (more than Claude!). I want it to be fast. I don't want it to be subsidized as that's just an excuse for shitty quality. Deepseek flash knocks it out of the park on all of these except you're data is used in training. I'm fine with it being hosted since there's no way I'm using it 24/7, but data MUST be private. Basically I want Hetzner and OVH to run open model clouds. I'm convinced this is going to happen eventually when everyone realizes this is a commodity.

Bnjoroge 4mo ago
You can specify which providers you want to serve your model in OpenRouter. Then you can chose US-based ones.
aamoscodes 4mo ago
You can pay, and also use deepseek-v4-flash. OpenRouter even lets you "block" or limit your usage to providers that don't train on data. Since the weights are open, other companies are already serving the model on non-DeepSeek owned hardware: https://openrouter.ai/deepseek/deepseek-v4-flash https://openrouter.ai/deepseek/deepseek-v4-flash
- rapind 4mo ago
  Good to know. I hadn't checks since early is DS4's launch when they were the only provide (I think maybe there was one other, but they also trained on your data). I see several private options now.
- fc417fc802 4mo ago
  > OpenRouter even lets you "block" or limit your usage to providers that don't train on data. More than that, they have various zero data retention options and provide a convenient json list of them.
  - larodi 4mo ago
    The fact OpenRouter strips https to reroute screams danger already.
    - fc417fc802 4mo ago
      What do you mean? Are you objecting that they communicate with the provider on your behalf? But how else would you design such a system? Plumbing you straight through would require nonstandard certificate juggling and they wouldn't be able to implement their core service of providing a standardized API nor could they transparently route your request to the fastest / cheapest / whatever provider on the fly nor could they implement transparent fallback nor could they implement their policy of not billing you if the response from the provider is invalid. Also the chosen provider could fingerprint your network stack if you communicated directly. The routing service is acting as a proxy and for most providers fully anonymizes requests (it does send a stable uid to some of them though).

spockz 4mo ago

For what it is worth, I’m on a similar machine. (9070XT,5900X) and found a lot of performance improvement over ollama by compiling llama.cpp and running with —no-mmap and —perf. The context is still quite small though. With online models I use contexts of at least 200k which is useful for longer running/more complicated commands. Locally I haven’t gone much further than 8k. That is sufficient for small changes on small code bases. And you need condensed tool output. I haven’t tried any tool that compresses the tokens yet.

echelon 4mo ago
I would rather we give up the idea of running open models on RTX cards and instead focus on running much bigger open models on H200s. 1. The hardware will eventually catch up. 2. This keeps the delta between frontier models smaller. 3. We can still fine tune and own the weights. 4. The models will be more useful, faster, and reliable. RTX is hobbyist tier, not professional tier. Gated cloud models from hyperscalers treat us like hobbyists in their own right. We need equivalent scale models, but open.
- MrLeap 4mo ago
  There's a lot more professionals that have RTX cards than H200s. You're inevitably see more development and experimentation on things actual humans have lmao.
- SR2Z 4mo ago
  That GPU costs 25k which means you really should have a rack to put it in. It's not realistic.
- zozbot234 4mo ago
  H200s and other enterprise datacenter GPUs are completely overkill in any realistic single- or few-users inference scenario. They're hugely unbalanced towards compute capacity which will go almost entirely unused (i.e. wasted) unless you're running huge batches on a continued basis. I've argued many times that local inference engines should support batched inference on a somewhat smaller scale for a variety of reasons (especially given the unexpected effectiveness of SSD streamed inference with larger-than-RAM models), but even I don't think we can realistically go to 300x or so for real-time inference, which is the range that pencils out quite consistently from a simple roofline model of these datacenter cards.

ryukoposting 4mo ago

I found that, with the heavily quantized Qwen3 models I can cram onto my 3060 Ti, telling the model to use its tools in the system prompt made it a lot more likely to actually do it. YMMV of course, but give it a shot.

saghm 4mo ago
I did try this, and it was pretty hit-or-miss still. I even went as far as configuring context for Zed to inject into all conversations saying stuff like "If you need to read a file, call read_file NOW. Do not say you will read it", and it still didn't really make a huge difference.

redmalang 4mo ago

Try llama.cpp it seems to be a lot more performant and a lot more hackable. Also I'm surprised how substantial the impact of some of the inference configs (beyond just temp) can have, though this is much more model specific.

boppo1 4mo ago

I have almost your system specs, how do they work for non-coding stuff like chat/knowledge/discussion? I've been using models to talk through social stuff I'm anxious about but dont want to annoy my friends with and it's been amazing, but I don't want to share that info with google/openai/anthropic anymore. I shouldn't have in the first place, but I couldn't help it, the exercise was too interesting.

fc417fc802 4mo ago
You can test the open models for yourself using the various router services. Those also make it easy to use providers other than the major players.

markussss 4mo ago

My system is quite similar to your, my GPU is a 6950 XT and CPU a Ryzen 5 2600x, same amount of RAM, and I feel your pain. It sounds very similar to my experience from a few months ago. When it comes to tool calling, there are multiple possible issues; some models have borked templates bundled with the model file, some models are not trained on tool calling, some agent harnesses doesn't support the tool call output from the model very well, some quantizations ruin the models' abilities to call tools. My suggestions if you want to further experiment with local models are to use llama.cpp instead of ollama [1], learn a little about the parameters that tune how much VRAM is used [2], look online for jinja template fixes for the model you're testing [3], and choose a model that was designed to do the task you want to achieve, with as high quantization as you can fit. The maximum model size you can run is VRAM + RAM, although you want as little of the model to be in system RAM as possible. I'm running North Mini Code IQ3_XXS with some tuned parameters to fit my current tasks, and while it is not perfect for everything, it has not failed any tool calls I've asked it to make, or that it figured it should make on its own. [1]: https://sleepingrobots.com/dreams/stop-using-ollama/ https://sleepingrobots.com/dreams/stop-using-ollama/ [2]: https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md https://github.com/ggml-org/llama.cpp/blob/master/tools/serv... [3]: https://gist.github.com/jscott3201/e4b155885cc68c038d6ac8909a3bd9fe https://gist.github.com/jscott3201/e4b155885cc68c038d6ac8909...

calgoo 4mo ago
"My suggestions if you want to further experiment with local models are to use llama.cpp instead of ollama" Or at least LM Studio if you want to play around with a lot of different models. Im currently using it with my 7800xt and Vulcan as i found it left my OS more stable ROCm does. I had a few system crashes with ROCm and running out of VRAM for the OS.
itissid 4mo ago
Interesting. Making low latency correct tool calls correctly is pretty important in voice AI cascading models(STT LLM TTS). Realtime Models are still 2x the cost and there are only 2 providers openai and google that are in the race. For cost control it has to be cascading models For llms Sadly the only model right now that fits the bill for LLM is GPT 4.1 and it’s standard in my stack because thinking models have unacceptable latency(>=1 sec) even though they are good at tool calling. The main issue with 4.1 is that it can make still mistakes and prompt prose has to be tuned quite a bit. I wonder if any local models can be tuned to match the response time and tool calling while supporting many languages.

alfiedotwtf 4mo ago

> qwen refused to believe that it was running in ollama and insisted that it was running from the Alibaba cloud without access to my local system Sounds like you were either running at a too-low quant, or you were trying to do Agents with something like Qwen 3.5 9B? Qwen 3.6 27B at Q4_K_M I can have that at running all night after a single one-shot a, Anne when I come back in the morning, it’s done