5 ms·

I ran the Q4 quant (used with llama.cpp) though my "minesweeper" vibe-coding benchmark: https://senko.net/vibecode-bench/2026/minesweeper-gamma-4-12b-q4.html ht

by senko 4mo ago

I ran the Q4 quant (used with llama.cpp) though my "minesweeper" vibe-coding benchmark: https://senko.net/vibecode-bench/2026/minesweeper-gamma-4-12b-q4.html https://senko.net/vibecode-bench/2026/minesweeper-gamma-4-12... The result is decent, but it had a few bizzare/trivial syntax errors I had to fix manually: it would do an extra closing bracket or paren a few times, and wanted to separate function definitions with comma. Not sure what that was about, but otherwise the output run just fine. So, with those qualifiers, I think it's a decent local coding model. It roughly compares with GPT-4.1 (!!), released 14 months ago, on the output: https://senko.net/vibecode-bench/2025/minesweeper-gpt-4.1.html https://senko.net/vibecode-bench/2025/minesweeper-gpt-4.1.ht... (actually I'd call it better, but those syntax errors...) I ran the quantized version (4-bit GGUF) on my consumer-grade card with 12G of VRAM and got 5t/s for output. Not for interactive use for coding, but fairly capable model. To me, it's fascinating how much progress we got in over a year. GPT-4.1 was considered an extremely capable coding model. Now we got something with 12B of params performing roughly the same (in this specific benchmark, disclaimers, etc). Lists of various models I tested: https://senko.net/vibecode-bench/ https://senko.net/vibecode-bench/

frikk 4mo ago
Thank you for sharing this. Do you think the syntactical issues could be addressed with fine tuning or some other kind of parameter tweaking? That's frustrating hah.
- profunctor 4mo ago
  With a harness you could feed the code to a linter and if there are errors feed that to a model automatically. It’s amazing that the models are good enough that I haven’t bothered doing this
zigzag312 4mo ago
> It roughly compares with GPT-4.1 (!!), released 14 months ago I think the mayor win for coding was reasoning. That's why such a small model can match GPT-4.1 in coding, but I suspect that GPT-4.1 still wins in general world knowledge due to bigger size.
- mdp2021 4mo ago
  > I suspect ... still wins in general world knowledge due to bigger size Encyclopedic knowledge matters relatively little in perspective, given the expectable future developments: even the more knowledgeable of us will use that knowledge for reasoning and intuition (and we will have absorbed the intellectual keys during our training), but under our professional hat we should in theory be ready to go "I stand corrected" and "more precisely" with the actual data at hand. I.e.: for the encyclopedic knowledge needed, the /understander/ will have a RAG subsystem and a corpus of knowledge to inquire upon processing queries. (Corroboration: we can't delirate, and neither can the machine...)
  - coldcity_again 4mo ago
    A great position to take. Strong opinions, weakly held.
  - bitexploder 4mo ago
    Don't LLMs work on attention though? The closer in their hyperdimensional space you can land your problem to their inherent understand the better they are at understanding your problem domain. RAG loops can be very slow and agents may simply lack the knowledge to use them correctly.
    - mdp2021 4mo ago
      But, in short, the ability to manage information, to process it properly, is more important in this regard than just having the information. "Having" more knowledge is not a guarantee to "using" it better. And to improve reliability, if the machine can check, it will have to check. "Costly" cannot be an excuse.
      - zigzag312 4mo ago
        Understanding of a specific problem space can be a prerequisite to be able to form a proper query (i.e. to ask the correct question). Model doesn't know what it doesn't know.
McGlockenshire 4mo ago
> my consumer-grade card with 12G of VRAM and got 5t/s for output Thank you for giving me hope!
0xbadcafebee 4mo ago
It was almost certainly not trained for coding, as it's got both audio and vision input, is only 12B, and nowhere in the announcement is coding mentioned. It will likely not have good performance on coding in general, compared to other small models like Qwen 3.6 35B A3B, Gemma 4 26B A4B, Nvidia Nemotron 3 Nano 30B-A3B, gpt-oss-20b. For 16GB laptops, Qwen 3.5 9B is the undisputed champ. Gemma 4 31B is the top dog at small model coding, but is dense so it needs ~48GB unified RAM for full context. If you want decent coding on a laptop you need a lot of RAM. But this shouldn't be surprising, dev machines have always needed lots of resources.
- senko 4mo ago
  Yeah, I agree 24B-36B sizes are better in general. I don't have unified RAM tho and offloading to CPU is dog slow, which is why I'm interested in 7b-12b models.
- kajecounterhack 4mo ago
  Have you found Gemma 4 31B better than Qwen 3.6 27B Q8? I just started using Qwen + Pi agent and it's great, but "which model works best" is still totally crowdsourced and I was going off of peoples' opinions on reddit. Would love to hear more opinions if people have them.
  - thangalin 4mo ago
    Yes. I'm using Gemma-4 31B (gemma-4-31B-it-assistant.Q4_K_M.gguf) with llama.cpp to attribute quotations throughout chapters of my sci-fi novel. I started with Qwen3, but couldn't get it to work. Qwen3 TTS Voice Design, on the other hand, is incredible (Qwen3-TTS-12Hz-1.7B-VoiceDesign). I'm using both for an audiobook generator that produces a variety of voices. Screens: * https://i.ibb.co/TBBV5nJk/kl-01.png https://i.ibb.co/TBBV5nJk/kl-01.png (voice design) * https://i.ibb.co/nNvvKDyV/kl-02.png https://i.ibb.co/nNvvKDyV/kl-02.png (quotation attributions)
    - khimaros 4mo ago
      building something similar: https://github.com/khimaros/autiobook https://github.com/khimaros/autiobook
superkuh 4mo ago
>consumer-grade card with 12G of VRAM and got 5t/s That speed for token output indicates to me that it somehow is using hybrid mode and involving cpu+system ram somehow. That ~5tk/s is about the ram bandwidth of DDR4 RAM versus that size model at 4bit. Any consumer GPU with 12 GB like a nvidia rtx 2080 or rtx 3060 should be doing 20+ tk/s with llama.cpp and CUDA backend.
- senko 4mo ago
  Good catch. I haven't looked deeply into it. This is with Vulkan backend on Linux which I understand should be roughly comparable to CUDA? Gfx is rtx 3060(ti?). I should play a bit more with llama.cpp options and see what bappened there. Thanks!
  - superkuh 4mo ago
    I've had it happen in the past with llama.cpp on linux that the CPU will present itself as a vulkan device GPU1 with "PHYSICAL_DEVICE_TYPE_CPU" and had a mix-up. Might want to try llama-server --list-devices and then append --device Vulkan0 or whatever.
- pja 4mo ago
  The 8 bit quant runs at 36tps using Vulkan on my AMD rx9070.
pseudosavant 4mo ago
Models this small and this capable bode really well for the usefulness of a PC like the RTX Spark that Nvidia/Microsoft announced this week. 128GB of unified memory will likely be more than sufficient for effective local agentic coding, even if SOTA cloud models will still be even better. Up until this point, I've found the cost/value to unequivocally favor using a cloud subscription, but I would be lying if I didn't worry that one day OpenAI is going to increase the price for my subscription by 5-10x. I rely on these tools enough that if there is a real viable local option, I'm going to take it.
- pseudollm 4mo ago
  > usefulness of the RTX Spark Not really. There's a reason the announcement didn't include ANY benchmark (!) and didn't mention EXACTLY what is the memory bandwidth. It's going to be dog-slow unusable for large models, as tok/sec is basically bandwidth divided by active weights. Rumoured 300GB/s / 30GB active weights (decent model) = 10 tokens per second, which is really slow
  - SwellJoe 4mo ago
    Yep, I have a Strix Halo and while it can run models bigger than Qwen 3.6 27b, it's not usable interactively when you do. ds4 patched for ROCm works, but at such a slow speed, it's not usable for coding agents. The Nvidia boxes have only slightly more memory bandwidth, so I wouldn't expect them to be notably faster. At least not enough to make it useful interactively at that scale.
    - zozbot234 4mo ago
      Why does everyone expect interactivity from local AI? It's not the best use of the hardware, especially not miniPC hardware. Long-term batched inference with larger and more capable models is much more feasible AIUI.
      - SwellJoe 4mo ago
        Even batched it's uncomfortably slow. I started to benchmark ds4 with my security vulnerability benchmark (after Qwen 3.6 dense and MoE and a bunch of cloud models), but it was going to tie up the Strix Halo for more than a day, so I decided not to run it as it would prevent me from doing other stuff with it during that time. Even batched usage needs to be fast enough to deliver results in a reasonable time. Overnight runs are useful, 24 hour runs are...less so. Anyway, most of the time people are talking about interactive use, and there's currently an upper bound on how large a model can be for local hosting on a reasonable budget (i.e. not a crazy amount more expensive than what a high end developer desktop or laptop costs). The sweet spot is probably currently the big Qwen 3.6 or Gemma 4 models, which are in the ~60GB range for 8-bit quantization plus a large context.
DeathArrow 4mo ago
>The result is decent, but it had a few bizzare/trivial syntax errors I had to fix manually Can you instruct it to use a lsp?
borissk 4mo ago
We are really getting close to singularity - the pace of LLM improvement is constantly accelerating.
UncleOxidant 4mo ago
I've heard the assertion that the Gemma 4 models don't do well with lower quantization. I wonder if the "bizzare/trivial" syntax errors would go away at Q8?
amelius 4mo ago
> it would do an extra closing bracket or paren a few times I had this with Gemini: in the middle of a C++ program it once said RParen instead of using ) It was easy to fix of course, but it makes you question what is going on inside its head.
pja 4mo ago
The Unsloth 8bit quant seems to manage this task without any syntax errors.

by senko 4mo ago

frikk 4mo ago

Thank you for sharing this. Do you think the syntactical issues could be addressed with fine tuning or some other kind of parameter tweaking? That's frustrating hah.

profunctor 4mo ago
With a harness you could feed the code to a linter and if there are errors feed that to a model automatically. It’s amazing that the models are good enough that I haven’t bothered doing this

zigzag312 4mo ago

> It roughly compares with GPT-4.1 (!!), released 14 months ago I think the mayor win for coding was reasoning. That's why such a small model can match GPT-4.1 in coding, but I suspect that GPT-4.1 still wins in general world knowledge due to bigger size.

mdp2021 4mo ago
> I suspect ... still wins in general world knowledge due to bigger size Encyclopedic knowledge matters relatively little in perspective, given the expectable future developments: even the more knowledgeable of us will use that knowledge for reasoning and intuition (and we will have absorbed the intellectual keys during our training), but under our professional hat we should in theory be ready to go "I stand corrected" and "more precisely" with the actual data at hand. I.e.: for the encyclopedic knowledge needed, the /understander/ will have a RAG subsystem and a corpus of knowledge to inquire upon processing queries. (Corroboration: we can't delirate, and neither can the machine...)
- coldcity_again 4mo ago
  A great position to take. Strong opinions, weakly held.
- bitexploder 4mo ago
  Don't LLMs work on attention though? The closer in their hyperdimensional space you can land your problem to their inherent understand the better they are at understanding your problem domain. RAG loops can be very slow and agents may simply lack the knowledge to use them correctly.
  - mdp2021 4mo ago
    But, in short, the ability to manage information, to process it properly, is more important in this regard than just having the information. "Having" more knowledge is not a guarantee to "using" it better. And to improve reliability, if the machine can check, it will have to check. "Costly" cannot be an excuse.
    - zigzag312 4mo ago
      Understanding of a specific problem space can be a prerequisite to be able to form a proper query (i.e. to ask the correct question). Model doesn't know what it doesn't know.

McGlockenshire 4mo ago

> my consumer-grade card with 12G of VRAM and got 5t/s for output Thank you for giving me hope!

0xbadcafebee 4mo ago

It was almost certainly not trained for coding, as it's got both audio and vision input, is only 12B, and nowhere in the announcement is coding mentioned. It will likely not have good performance on coding in general, compared to other small models like Qwen 3.6 35B A3B, Gemma 4 26B A4B, Nvidia Nemotron 3 Nano 30B-A3B, gpt-oss-20b. For 16GB laptops, Qwen 3.5 9B is the undisputed champ. Gemma 4 31B is the top dog at small model coding, but is dense so it needs ~48GB unified RAM for full context. If you want decent coding on a laptop you need a lot of RAM. But this shouldn't be surprising, dev machines have always needed lots of resources.

senko 4mo ago
Yeah, I agree 24B-36B sizes are better in general. I don't have unified RAM tho and offloading to CPU is dog slow, which is why I'm interested in 7b-12b models.
kajecounterhack 4mo ago
Have you found Gemma 4 31B better than Qwen 3.6 27B Q8? I just started using Qwen + Pi agent and it's great, but "which model works best" is still totally crowdsourced and I was going off of peoples' opinions on reddit. Would love to hear more opinions if people have them.
- thangalin 4mo ago
  Yes. I'm using Gemma-4 31B (gemma-4-31B-it-assistant.Q4_K_M.gguf) with llama.cpp to attribute quotations throughout chapters of my sci-fi novel. I started with Qwen3, but couldn't get it to work. Qwen3 TTS Voice Design, on the other hand, is incredible (Qwen3-TTS-12Hz-1.7B-VoiceDesign). I'm using both for an audiobook generator that produces a variety of voices. Screens: * https://i.ibb.co/TBBV5nJk/kl-01.png https://i.ibb.co/TBBV5nJk/kl-01.png (voice design) * https://i.ibb.co/nNvvKDyV/kl-02.png https://i.ibb.co/nNvvKDyV/kl-02.png (quotation attributions)
  - khimaros 4mo ago
    building something similar: https://github.com/khimaros/autiobook https://github.com/khimaros/autiobook

superkuh 4mo ago

>consumer-grade card with 12G of VRAM and got 5t/s That speed for token output indicates to me that it somehow is using hybrid mode and involving cpu+system ram somehow. That ~5tk/s is about the ram bandwidth of DDR4 RAM versus that size model at 4bit. Any consumer GPU with 12 GB like a nvidia rtx 2080 or rtx 3060 should be doing 20+ tk/s with llama.cpp and CUDA backend.

senko 4mo ago
Good catch. I haven't looked deeply into it. This is with Vulkan backend on Linux which I understand should be roughly comparable to CUDA? Gfx is rtx 3060(ti?). I should play a bit more with llama.cpp options and see what bappened there. Thanks!
- superkuh 4mo ago
  I've had it happen in the past with llama.cpp on linux that the CPU will present itself as a vulkan device GPU1 with "PHYSICAL_DEVICE_TYPE_CPU" and had a mix-up. Might want to try llama-server --list-devices and then append --device Vulkan0 or whatever.
pja 4mo ago
The 8 bit quant runs at 36tps using Vulkan on my AMD rx9070.

pseudosavant 4mo ago

Models this small and this capable bode really well for the usefulness of a PC like the RTX Spark that Nvidia/Microsoft announced this week. 128GB of unified memory will likely be more than sufficient for effective local agentic coding, even if SOTA cloud models will still be even better. Up until this point, I've found the cost/value to unequivocally favor using a cloud subscription, but I would be lying if I didn't worry that one day OpenAI is going to increase the price for my subscription by 5-10x. I rely on these tools enough that if there is a real viable local option, I'm going to take it.

pseudollm 4mo ago
> usefulness of the RTX Spark Not really. There's a reason the announcement didn't include ANY benchmark (!) and didn't mention EXACTLY what is the memory bandwidth. It's going to be dog-slow unusable for large models, as tok/sec is basically bandwidth divided by active weights. Rumoured 300GB/s / 30GB active weights (decent model) = 10 tokens per second, which is really slow
- SwellJoe 4mo ago
  Yep, I have a Strix Halo and while it can run models bigger than Qwen 3.6 27b, it's not usable interactively when you do. ds4 patched for ROCm works, but at such a slow speed, it's not usable for coding agents. The Nvidia boxes have only slightly more memory bandwidth, so I wouldn't expect them to be notably faster. At least not enough to make it useful interactively at that scale.
  - zozbot234 4mo ago
    Why does everyone expect interactivity from local AI? It's not the best use of the hardware, especially not miniPC hardware. Long-term batched inference with larger and more capable models is much more feasible AIUI.
    - SwellJoe 4mo ago
      Even batched it's uncomfortably slow. I started to benchmark ds4 with my security vulnerability benchmark (after Qwen 3.6 dense and MoE and a bunch of cloud models), but it was going to tie up the Strix Halo for more than a day, so I decided not to run it as it would prevent me from doing other stuff with it during that time. Even batched usage needs to be fast enough to deliver results in a reasonable time. Overnight runs are useful, 24 hour runs are...less so. Anyway, most of the time people are talking about interactive use, and there's currently an upper bound on how large a model can be for local hosting on a reasonable budget (i.e. not a crazy amount more expensive than what a high end developer desktop or laptop costs). The sweet spot is probably currently the big Qwen 3.6 or Gemma 4 models, which are in the ~60GB range for 8-bit quantization plus a large context.

DeathArrow 4mo ago

>The result is decent, but it had a few bizzare/trivial syntax errors I had to fix manually Can you instruct it to use a lsp?

borissk 4mo ago

We are really getting close to singularity - the pace of LLM improvement is constantly accelerating.

UncleOxidant 4mo ago

I've heard the assertion that the Gemma 4 models don't do well with lower quantization. I wonder if the "bizzare/trivial" syntax errors would go away at Q8?

amelius 4mo ago

> it would do an extra closing bracket or paren a few times I had this with Gemini: in the middle of a C++ program it once said RParen instead of using ) It was easy to fix of course, but it makes you question what is going on inside its head.

pja 4mo ago

The Unsloth 8bit quant seems to manage this task without any syntax errors.