3 ms·

It was actually great. I have like a non-AI box so to speak 8GB VRAM, co-incidentally from a gaming PC ... All the previous models that were "frontier level, j

It was actually great. I have like a non-AI box so to speak 8GB VRAM, co-incidentally from a gaming PC ... All the previous models that were "frontier level, just try it!" but wouldn't run at all in agentic mode, including previous Qwens, just disappointed, period. Then I ran then Qwen 3.8 27b and while it was super slow (4t/s) it literally one-shotted creating a usable "web search/pull" skill for `pi.dev`. while any other model previously just entirely failed to create anything usable even with actual guidance. Since then I have actually gotten a gemma-4 12B qat 4bit quantized with a ~250MB MTP from unsloth to work with a 32k context "working" on this setup at 80-120 t/s. That's usable for private stuff on a co-incidental box! It's still only 32k context and it's entirely dumb vs. our API paid at-work Claude Opus. But for entirely private local stuff it's totally workable without breaking the bank even after all these AI price hikes!. I bought this rig literally just for gaming a month ago.

aktenlage 1mo ago
Have you tried a mixture of experts model? Dense models have been quite slow for me, as I have only 6 GB VRAM. But with llama.cpp and --cpu-moe I get 200 t/s input and almost 30 t/s output with Gemma 4 26B A3B, which feels ok to use. Would be interested about your mileage there.
- b112 1mo ago
  I wish qwen3.8 had a MoE variant, but the skinny is it won't be coming.
- tharkun__ 1mo ago
  If I use the 12B Unified (dense) model I mentioned without MTP, then I get 37t/s, input ~700t/s. It's all still quite frustrating in the end, like a Claude from a very long time ago by now but usable. If I want 64k context, I can't use MTP. I still haven't decided whether I'd rather have 37t/s but it's "less dumb" or I want MTP speed but it's going off the rails more. All of this is also with `-ctv q4_0 -ctk q4_0)`, which is not ideal. I'm actually right now contending with 35k context but using q8_0 KV quantization. More like 35t/s coz with those settings I can't use MTP. But I'm not ready to go back to 4t/s. It's not interactive enough for me. That said, I had tried to use the Gemma E4B for example to have it build itself that websearch/fetch skill. It utterly failed, as did previous qwens. I don't see a Gemma 4 26B A3B GGUF for download, but there is a gemma-4-26B-A4B-it-MXFP4_MOE.gguf that should fit into my overall RAM and then use lots of CPU like the Qwen 3.8. I guess I'll give it a try just to see the difference in speed though I don't expect anything "usable" out of that tbh.
  - tharkun__ 1mo ago
    Haha, OK, so it processes input tokens at 55t/s if started like the 12B. That's so slow that I don't even have the patience to wait for the context of the session I aborted mid-processing to test this, to load. Heck loading the system prompt alone is not fun. That said, after that it seems to run at the same ~35t/s speed but, I was able to start it up with a 128k context (and no KV quantization), humm humm. I am now playing more with all the parameters like I did to get the 12B going e.g. I'm already up to ~125t/s for inputs now and with an 8bit MTP it's running about 45-85t/s output when its coding. I run with `--spec-type draft-mtp --spec-draft-p-min 0.8 --spec-draft-n-max 8` and when it was that fast (during actual code generation) got some `draft acceptance = 0.91801 ( 3482 accepted / 3793 generated), mean len = 7.92`. I also tried 16 with no real improvement for those phases but it seemed slower, wasting too much with low 50% acceptances during other phases like thinking. EDIT: Oh look, I can now use larger batch sizes again and that gets prompt_eval reads back to 700+ Regarding how good it is: Probably, maybe, better than the 12B? It immediately noticed an error the 12B made, where it hadn't followed a skill's format description. But it's also annoyingly not following instructions the other one was and instead just gives me its opinion lol! And it couldn't figure out how to read a simple pi SKILL.md and run an included script. It complained that it wasn't registered as a tool. But like all other model changes, will get used to it and adjust I suppose :)
    - aktenlage 29d ago
      Thank you, very interesting to read. I haven't considered batch sizes at all up to now. Seems I should look into it. How did you get the prediction that good? The default spec-draft-max-n is 3 and if I go beyond, the output rate stays the same or gets lower any magic spell you could share?
      - aktenlage 27d ago
        Wow, setting the "ubatch" size to 2048 (default is 512), like the "batch" size roughly doubled prompt processing speed.
      - tharkun__ 25d ago
big_babol 1mo ago
I'm having the same constraint, 8GB VRAM, and was trying the past months to get a local model running to drive my Home Assistant setup via voice, so many tool calls, little room for errors. I've tried multiple models, different temperature and top p values, different context windows... turns out that a quantized gemma4 is the best pick so far but qwen3.5:9b isn't far off. Surprisingly newer models are not necessarily performing better.
- kQq9oHeAz6wLLS 1mo ago
  I've had great success using Gemma4 even on old hardware (4GB VRAM). I haven't found a Qwen model that can match it on very low end hardware.

aktenlage 1mo ago

Have you tried a mixture of experts model? Dense models have been quite slow for me, as I have only 6 GB VRAM. But with llama.cpp and --cpu-moe I get 200 t/s input and almost 30 t/s output with Gemma 4 26B A3B, which feels ok to use. Would be interested about your mileage there.

b112 1mo ago
I wish qwen3.8 had a MoE variant, but the skinny is it won't be coming.
tharkun__ 1mo ago
If I use the 12B Unified (dense) model I mentioned without MTP, then I get 37t/s, input ~700t/s. It's all still quite frustrating in the end, like a Claude from a very long time ago by now but usable. If I want 64k context, I can't use MTP. I still haven't decided whether I'd rather have 37t/s but it's "less dumb" or I want MTP speed but it's going off the rails more. All of this is also with `-ctv q4_0 -ctk q4_0)`, which is not ideal. I'm actually right now contending with 35k context but using q8_0 KV quantization. More like 35t/s coz with those settings I can't use MTP. But I'm not ready to go back to 4t/s. It's not interactive enough for me. That said, I had tried to use the Gemma E4B for example to have it build itself that websearch/fetch skill. It utterly failed, as did previous qwens. I don't see a Gemma 4 26B A3B GGUF for download, but there is a gemma-4-26B-A4B-it-MXFP4_MOE.gguf that should fit into my overall RAM and then use lots of CPU like the Qwen 3.8. I guess I'll give it a try just to see the difference in speed though I don't expect anything "usable" out of that tbh.
- tharkun__ 1mo ago
  Haha, OK, so it processes input tokens at 55t/s if started like the 12B. That's so slow that I don't even have the patience to wait for the context of the session I aborted mid-processing to test this, to load. Heck loading the system prompt alone is not fun. That said, after that it seems to run at the same ~35t/s speed but, I was able to start it up with a 128k context (and no KV quantization), humm humm. I am now playing more with all the parameters like I did to get the 12B going e.g. I'm already up to ~125t/s for inputs now and with an 8bit MTP it's running about 45-85t/s output when its coding. I run with `--spec-type draft-mtp --spec-draft-p-min 0.8 --spec-draft-n-max 8` and when it was that fast (during actual code generation) got some `draft acceptance = 0.91801 ( 3482 accepted / 3793 generated), mean len = 7.92`. I also tried 16 with no real improvement for those phases but it seemed slower, wasting too much with low 50% acceptances during other phases like thinking. EDIT: Oh look, I can now use larger batch sizes again and that gets prompt_eval reads back to 700+ Regarding how good it is: Probably, maybe, better than the 12B? It immediately noticed an error the 12B made, where it hadn't followed a skill's format description. But it's also annoyingly not following instructions the other one was and instead just gives me its opinion lol! And it couldn't figure out how to read a simple pi SKILL.md and run an included script. It complained that it wasn't registered as a tool. But like all other model changes, will get used to it and adjust I suppose :)
  - aktenlage 29d ago
    Thank you, very interesting to read. I haven't considered batch sizes at all up to now. Seems I should look into it. How did you get the prediction that good? The default spec-draft-max-n is 3 and if I go beyond, the output rate stays the same or gets lower any magic spell you could share?
    - aktenlage 27d ago
      Wow, setting the "ubatch" size to 2048 (default is 512), like the "batch" size roughly doubled prompt processing speed.
    - tharkun__ 25d ago

big_babol 1mo ago

I'm having the same constraint, 8GB VRAM, and was trying the past months to get a local model running to drive my Home Assistant setup via voice, so many tool calls, little room for errors. I've tried multiple models, different temperature and top p values, different context windows... turns out that a quantized gemma4 is the best pick so far but qwen3.5:9b isn't far off. Surprisingly newer models are not necessarily performing better.

kQq9oHeAz6wLLS 1mo ago
I've had great success using Gemma4 even on old hardware (4GB VRAM). I haven't found a Qwen model that can match it on very low end hardware.