3 ms·

Gemma is better than Qwen at everything except coding, in all my evaluations. Which is a shame because that is what I use them for!

by girvo 4mo ago

UncleOxidant 4mo ago
It would be great if the Gemma folks would release a code-focused model. Probably won't happen, but it's fun to dream.
- SwellJoe 4mo ago
  The Ornith folks say they're doing that, but haven't released the Gemma-based 31b yet (https://github.com/deepreinforce-ai/Ornith-1 https://github.com/deepreinforce-ai/Ornith-1). But, also, the Qwen-based 35b MoE Ornith version performs worse than Qwen 3.6 and Qwen AgentWorld on my benchmarks (which are focused on finding security bugs, so not exactly the same as agentic coding, but closely related skills). That said, the reason they're able to release Ornith branded post-trains of both Gemma and Qwen is because they're open weights under a friendly license. Someone, not just Google, could make a coding focused Gemma post-train. I don't think it's actually much weaker than Qwen 3.6 for coding; Gemma 4 31b outperforms Qwen 3.6 27b by a wide margin on security bug hunting (at least for the specific bugs in my benchmarks, which are mostly relatively difficult bugs from the Mythos-reported bugs). I'd really love to see a bigger MoE from Google, though. A 70b or 120b MoE would likely be super fun.
  - automajicly 3mo ago
    I know this wasn't meant as a response to me, but I thought I would chime in, if thats okay. Sorry its a bit of time since the timing of this message.And I hope Im not intrusive. But to give you a bit of an update, I broke down and subbed to Claude's PRO feature. I know it defeats the purpose of local and private, but I still have my Gemma in my private LM studio. WOW. The coding diff is like night and day. I had Claude Code audit my repo page and it was quick and production grade. Like it systematically just chiseld away at all the sloppy choppy and dead cade and even polished up the aesthetics of my page presentation. I need this as a local private model.
- urbsgpw 4mo ago
  Ya, doesn't seem to be google's focus at all, right?
deleted 4mo ago
[deleted]
ekianjo 4mo ago
gemma is also worse for tool calling. not just coding
- satvikpendem 4mo ago
  That is because they use a different tool calling format than most other models. Unsloth quants fix this in their Gemma releases.
  - stevenhubertron 4mo ago
    Can you say a bit more about this? The bad tool calling has made me give up on using Gemma for my Hermes and a personal recipe site. I have only downloaded from Ollama.
    - satvikpendem 4mo ago
      Ollama is not recommended [0], use llama.cpp or more specifically Unsloth Studio which wraps llama.cpp and which has an API mode you can use to hook into Hermes or another agent. Unsloth make both the Studio and the quants which fix various issues with many models [1] as well as implementing new features like MTP and QAT support much sooner than other teams. In general you should read r/LocalLLaMa as it has a lot of updates regarding local models as the field moves fast. [0] https://sleepingrobots.com/dreams/stop-using-ollama/ https://sleepingrobots.com/dreams/stop-using-ollama/ [1] https://github.com/unslothai/unsloth/discussions/4921 https://github.com/unslothai/unsloth/discussions/4921
      - Der_Einzige 4mo ago
        Everytime I try to mention how shit Ollama is, I get mass downvoted here by folks that don't want to hear the truth: There are 4 good inference engines (okay, 5, but we don't count huggingface because it's slow): 1. vLLM 2. sglang 3. (nvidia only) TRT-LLM 4. llama.cpp (mac only, the above are better for non-mac) If you're not using one of the above, you're doing it wrong
  - 4mo ago
lambda 4mo ago
I haven't run a proper eval, but I've been getting better luck with Qwen models than Gemma on plant and animal identification using vision. I do like Gemma for translation, however.
- SwellJoe 4mo ago
  Gemma 12B? It's unique in the Gemma family, and unique among vision models. It's a novel encoder-less model...the whole model is vision. Somehow. I don't understand it, but it blows away Gemma 4 31B and Qwen 27B in my tests. It's not even close. And, is also tiny and fast, compared to those larger models, so it's better and faster and smaller. Weird combo.
  - lambda 4mo ago
    Tried it out. I'm compring against Qwen 3.5 122B-A10B, so a much larger model. It gets some correct, but Qwen 3.5 122B-A10B has done much better. Gemma 4 12B even hallucinated some species in trying to identify a plant, and the other guesses it made weren't all that close, while Qwen 3.5 122B-A10B got it right on the first try. 12B did get one right that 31B got wrong. I'd have to do a much more thorough eval to really compare, just a few anecdotal observations and it's kind of hard to really distinguish, but from the samples I've seen, Qwen 3.5 122B-A10B is doing much better at this task. The 12B architecture definitely is interesting, and it may punch above its weight due to this (though again, would really need to do proper evals to compare). But of the models I've tried, Qwen3.5 122B-A10B really seems like the best for this kind of task.
    - SwellJoe 3mo ago
      Ah, yeah, a model ten times as large on disk will know more, for sure. My tasks are more about "what's happening in this image?" rather than "what is this thing?", which doesn't require encyclopedic knowledge, it need good reasoning about what it's seeing.
automajicly 4mo ago
I have a M1 Macbook Pro...with only 16gb and I struggled with Qwens2.5-14b trying to do large projects. I loved Qwen but I had to try and do something different. So I switched to Gemma4-12b which looking at it now, seems more like a downgrade than an upgrade.Can you refer me to any Qwen coding models that wont choke my poor 16gb and also connect contextually? I need that context. I love the laser point focus, but I need context and basic understanding of that context.
- SwellJoe 4mo ago
  I don't think "large projects" is realistic with a model that fits in ~8GB (I'm assuming you run stuff other than the model). And, Gemma 4 12B QAT at 4-bits is surely the smartest model in its size, but it shines at vision tasks rather than agentic tasks (though it is a good tool user and can do stuff like research, it's obviously not aimed at code). You can almost always find free models on OpenRouter. Google AI Studio also has free usage of Gemma 4. Both are rate and usage limited which agentic use will probably chew up pretty quick, but you can usually find some pretty powerful models for free. If you rotate through different providers, I think it avoids the cap. Currently several Nemotron models, North Mini Code, Laguna models, Gemma 4 31b and MoE, Qwen 3 Next, and gpt-oss 120b, are all available free on OpenRouter...and better than anything you can run locally in ~8GB.
  - automajicly 4mo ago
    WOW. That was alot of models. I will be sure to look into those. What caught my eye was the North Mini Code. How is that at reason and context? I guess I will just go see. Thank you for the insight. I am building and learning at the same time, and I really need a model that wont chew up my poor little M1, but at the same time will understand my direction, my context, and also my explanation of need and then be ...' oh, I know what you mean..' then help vibe with me to create it. Qwen was a code model but hallucinated and was never really good at contextual understanding. Gemma seems to get it but isnt necessarily a coding model.(As you said) And I just cant be bogged down with all these fees to get what I need on my journey. So onward on the model search I suppose. Thank you very much for the time.

Gemma is better than Qwen at everything except coding, in all my evaluations. Which is a shame because that is what I use them for!

by girvo 4mo ago

Gemma is better than Qwen at everything except coding, in all my evaluations. Which is a shame because that is what I use them for!

UncleOxidant 4mo ago

It would be great if the Gemma folks would release a code-focused model. Probably won't happen, but it's fun to dream.

SwellJoe 4mo ago
The Ornith folks say they're doing that, but haven't released the Gemma-based 31b yet (https://github.com/deepreinforce-ai/Ornith-1 https://github.com/deepreinforce-ai/Ornith-1). But, also, the Qwen-based 35b MoE Ornith version performs worse than Qwen 3.6 and Qwen AgentWorld on my benchmarks (which are focused on finding security bugs, so not exactly the same as agentic coding, but closely related skills). That said, the reason they're able to release Ornith branded post-trains of both Gemma and Qwen is because they're open weights under a friendly license. Someone, not just Google, could make a coding focused Gemma post-train. I don't think it's actually much weaker than Qwen 3.6 for coding; Gemma 4 31b outperforms Qwen 3.6 27b by a wide margin on security bug hunting (at least for the specific bugs in my benchmarks, which are mostly relatively difficult bugs from the Mythos-reported bugs). I'd really love to see a bigger MoE from Google, though. A 70b or 120b MoE would likely be super fun.
- automajicly 3mo ago
  I know this wasn't meant as a response to me, but I thought I would chime in, if thats okay. Sorry its a bit of time since the timing of this message.And I hope Im not intrusive. But to give you a bit of an update, I broke down and subbed to Claude's PRO feature. I know it defeats the purpose of local and private, but I still have my Gemma in my private LM studio. WOW. The coding diff is like night and day. I had Claude Code audit my repo page and it was quick and production grade. Like it systematically just chiseld away at all the sloppy choppy and dead cade and even polished up the aesthetics of my page presentation. I need this as a local private model.
urbsgpw 4mo ago
Ya, doesn't seem to be google's focus at all, right?

deleted 4mo ago

[deleted]

ekianjo 4mo ago

gemma is also worse for tool calling. not just coding

satvikpendem 4mo ago
That is because they use a different tool calling format than most other models. Unsloth quants fix this in their Gemma releases.
- stevenhubertron 4mo ago
  Can you say a bit more about this? The bad tool calling has made me give up on using Gemma for my Hermes and a personal recipe site. I have only downloaded from Ollama.
  - satvikpendem 4mo ago
    Ollama is not recommended [0], use llama.cpp or more specifically Unsloth Studio which wraps llama.cpp and which has an API mode you can use to hook into Hermes or another agent. Unsloth make both the Studio and the quants which fix various issues with many models [1] as well as implementing new features like MTP and QAT support much sooner than other teams. In general you should read r/LocalLLaMa as it has a lot of updates regarding local models as the field moves fast. [0] https://sleepingrobots.com/dreams/stop-using-ollama/ https://sleepingrobots.com/dreams/stop-using-ollama/ [1] https://github.com/unslothai/unsloth/discussions/4921 https://github.com/unslothai/unsloth/discussions/4921
    - Der_Einzige 4mo ago
      Everytime I try to mention how shit Ollama is, I get mass downvoted here by folks that don't want to hear the truth: There are 4 good inference engines (okay, 5, but we don't count huggingface because it's slow): 1. vLLM 2. sglang 3. (nvidia only) TRT-LLM 4. llama.cpp (mac only, the above are better for non-mac) If you're not using one of the above, you're doing it wrong
- 4mo ago

lambda 4mo ago

I haven't run a proper eval, but I've been getting better luck with Qwen models than Gemma on plant and animal identification using vision. I do like Gemma for translation, however.

SwellJoe 4mo ago
Gemma 12B? It's unique in the Gemma family, and unique among vision models. It's a novel encoder-less model...the whole model is vision. Somehow. I don't understand it, but it blows away Gemma 4 31B and Qwen 27B in my tests. It's not even close. And, is also tiny and fast, compared to those larger models, so it's better and faster and smaller. Weird combo.
- lambda 4mo ago
  Tried it out. I'm compring against Qwen 3.5 122B-A10B, so a much larger model. It gets some correct, but Qwen 3.5 122B-A10B has done much better. Gemma 4 12B even hallucinated some species in trying to identify a plant, and the other guesses it made weren't all that close, while Qwen 3.5 122B-A10B got it right on the first try. 12B did get one right that 31B got wrong. I'd have to do a much more thorough eval to really compare, just a few anecdotal observations and it's kind of hard to really distinguish, but from the samples I've seen, Qwen 3.5 122B-A10B is doing much better at this task. The 12B architecture definitely is interesting, and it may punch above its weight due to this (though again, would really need to do proper evals to compare). But of the models I've tried, Qwen3.5 122B-A10B really seems like the best for this kind of task.
  - SwellJoe 3mo ago
    Ah, yeah, a model ten times as large on disk will know more, for sure. My tasks are more about "what's happening in this image?" rather than "what is this thing?", which doesn't require encyclopedic knowledge, it need good reasoning about what it's seeing.

automajicly 4mo ago

I have a M1 Macbook Pro...with only 16gb and I struggled with Qwens2.5-14b trying to do large projects. I loved Qwen but I had to try and do something different. So I switched to Gemma4-12b which looking at it now, seems more like a downgrade than an upgrade.Can you refer me to any Qwen coding models that wont choke my poor 16gb and also connect contextually? I need that context. I love the laser point focus, but I need context and basic understanding of that context.

SwellJoe 4mo ago
I don't think "large projects" is realistic with a model that fits in ~8GB (I'm assuming you run stuff other than the model). And, Gemma 4 12B QAT at 4-bits is surely the smartest model in its size, but it shines at vision tasks rather than agentic tasks (though it is a good tool user and can do stuff like research, it's obviously not aimed at code). You can almost always find free models on OpenRouter. Google AI Studio also has free usage of Gemma 4. Both are rate and usage limited which agentic use will probably chew up pretty quick, but you can usually find some pretty powerful models for free. If you rotate through different providers, I think it avoids the cap. Currently several Nemotron models, North Mini Code, Laguna models, Gemma 4 31b and MoE, Qwen 3 Next, and gpt-oss 120b, are all available free on OpenRouter...and better than anything you can run locally in ~8GB.
- automajicly 4mo ago
  WOW. That was alot of models. I will be sure to look into those. What caught my eye was the North Mini Code. How is that at reason and context? I guess I will just go see. Thank you for the insight. I am building and learning at the same time, and I really need a model that wont chew up my poor little M1, but at the same time will understand my direction, my context, and also my explanation of need and then be ...' oh, I know what you mean..' then help vibe with me to create it. Qwen was a code model but hallucinated and was never really good at contextual understanding. Gemma seems to get it but isnt necessarily a coding model.(As you said) And I just cant be bogged down with all these fees to get what I need on my journey. So onward on the model search I suppose. Thank you very much for the time.