3 ms·

Can you please tell which Gemma 4 variant managed to correctly reason through your private benchmarks? Was is Gemma 4 31B? What quantizations and context lengt

by pulse7 2mo ago

Can you please tell which Gemma 4 variant managed to correctly reason through your private benchmarks? Was is Gemma 4 31B? What quantizations and context lengths did you use for Gemma 4 and Qwen 3.8 27B? I am asking because I can't even load Gemma 4 31B on my GPU with any reasonable quantization (even with small context), while I can run Qwen 3.8 27B with large context and good quantization...

jodleif 2mo ago
Then you might be missing SWA. Gemma models are extremely memory hungry without
- CMay 2mo ago
  So long as they have flash attention enabled, Llama.cpp enables Sliding Window Attention by default for Gemma 4 models. Even if they're using Ollama or LM Studio I would expect those to mostly be doing the right things.
  - DiabloD3 2mo ago
    I would not expect Ollama to be doing the right thing fwiw.
CMay 2mo ago
Gemma 4 12B, Gemma 4 12B QAT, Gemma 4 31B, Gemma 4 31B QAT Gemma 4 26BA4B would get close, but not quite and sometimes even get stuck in loops despite a repeat penalty. Do not use any newer updated templates or Unsloth fixes. Use older official templates that released with the models on the huggingface repo. The template here worked: https://huggingface.co/google/gemma-4-12B-it/tree/657684fef0b5ac5d6bff39284ceb6ec3710b700e https://huggingface.co/google/gemma-4-12B-it/tree/657684fef0... llama-server --model "model.gguf" -fa on -np 1 --jinja --ctx-size 262144 -b 768 -ub 768 --cache-type-k f16 --cache-type-v q4_0 --repeat-penalty 1.1 --chat-template-file "chat_template.jinja" If you don't explicitly point to the template file, then llama.cpp will either use the template inside the model file or it will use its own template copy and your results may vary. Obviously some of the template fixes are useful to people, so it depends if you're having problems with tool calling or can't fix the tool calling in other ways for your scenario. My experience with the QAT models was that quantizing v to q4_0 gave me better results than q8_0 or even f16. I think the Gemma QAT models may have been QAT trained to expect a q4_0 quantized v cache. If you're not using a QAT model, I would leave both at f16. Another thing aside from using the QAT models and a Q4_0 v cache since you're having trouble fitting the models, is that you don't have to use the mmproj if you don't intend to use vision. If you need vision, but are hurting on VRAM, then you should be using --no-mmproj-offload. That will keep the mmproj loaded in system RAM instead of on your GPU. Loading images will be a little bit slower, but it can still be quite fast and you'll have more breathing room on your GPU. If you don't provide the mmproj file on the command line at all, then it won't load it anyway. If you're using some program like LM Studio, a simple thing you can do is move the mmproj and mtp files out of the directory for the model so LM studio can't find them and then it won't load them at all. For Qwen 3.8 27B, doing any quantizing definitely hurt results a lot, so in my case I used: llama-server --model "Qwen3.8-27B-UD-Q4_K_XL.gguf" --spec-type draft-mtp --spec-draft-p-min 0.35 --spec-draft-n-max 2 -fa on -np 1 --jinja --ctx-size 65536 -b 768 -ub 768 --cache-type-k f16 --cache-type-v f16
- kpatucha 2mo ago
  Gemma 4 12B? This sounds really interesting with Q_4 (preferably QAT) this fits comfortably in 12 or 16 GB VRAM. Could you elaborate on Gemma 4 12B capabilities from your experience and benchmarks?

by pulse7 2mo ago

jodleif 2mo ago

Then you might be missing SWA. Gemma models are extremely memory hungry without

CMay 2mo ago
So long as they have flash attention enabled, Llama.cpp enables Sliding Window Attention by default for Gemma 4 models. Even if they're using Ollama or LM Studio I would expect those to mostly be doing the right things.
- DiabloD3 2mo ago
  I would not expect Ollama to be doing the right thing fwiw.

CMay 2mo ago

Gemma 4 12B, Gemma 4 12B QAT, Gemma 4 31B, Gemma 4 31B QAT Gemma 4 26BA4B would get close, but not quite and sometimes even get stuck in loops despite a repeat penalty. Do not use any newer updated templates or Unsloth fixes. Use older official templates that released with the models on the huggingface repo. The template here worked: https://huggingface.co/google/gemma-4-12B-it/tree/657684fef0b5ac5d6bff39284ceb6ec3710b700e https://huggingface.co/google/gemma-4-12B-it/tree/657684fef0... llama-server --model "model.gguf" -fa on -np 1 --jinja --ctx-size 262144 -b 768 -ub 768 --cache-type-k f16 --cache-type-v q4_0 --repeat-penalty 1.1 --chat-template-file "chat_template.jinja" If you don't explicitly point to the template file, then llama.cpp will either use the template inside the model file or it will use its own template copy and your results may vary. Obviously some of the template fixes are useful to people, so it depends if you're having problems with tool calling or can't fix the tool calling in other ways for your scenario. My experience with the QAT models was that quantizing v to q4_0 gave me better results than q8_0 or even f16. I think the Gemma QAT models may have been QAT trained to expect a q4_0 quantized v cache. If you're not using a QAT model, I would leave both at f16. Another thing aside from using the QAT models and a Q4_0 v cache since you're having trouble fitting the models, is that you don't have to use the mmproj if you don't intend to use vision. If you need vision, but are hurting on VRAM, then you should be using --no-mmproj-offload. That will keep the mmproj loaded in system RAM instead of on your GPU. Loading images will be a little bit slower, but it can still be quite fast and you'll have more breathing room on your GPU. If you don't provide the mmproj file on the command line at all, then it won't load it anyway. If you're using some program like LM Studio, a simple thing you can do is move the mmproj and mtp files out of the directory for the model so LM studio can't find them and then it won't load them at all. For Qwen 3.8 27B, doing any quantizing definitely hurt results a lot, so in my case I used: llama-server --model "Qwen3.8-27B-UD-Q4_K_XL.gguf" --spec-type draft-mtp --spec-draft-p-min 0.35 --spec-draft-n-max 2 -fa on -np 1 --jinja --ctx-size 65536 -b 768 -ub 768 --cache-type-k f16 --cache-type-v f16

kpatucha 2mo ago
Gemma 4 12B? This sounds really interesting with Q_4 (preferably QAT) this fits comfortably in 12 or 16 GB VRAM. Could you elaborate on Gemma 4 12B capabilities from your experience and benchmarks?