3 ms·

4bits is a cutoff point for many model families, but also depends on what parts you quant to 4bits vs alternatives (weights, weight+activation, kv cache). Also

4bits is a cutoff point for many model families, but also depends on what parts you quant to 4bits vs alternatives (weights, weight+activation, kv cache). Also depends on model size and task, lots of nuance in quanting I've come to learn. Good evaluation from 2024 https://arxiv.org/pdf/2402.18158 https://arxiv.org/pdf/2402.18158 I'm currently working towards an updated version (not an og author), curious if others are aware of similar surveys, as I have yet to do a real lit search.

dofm 4mo ago
The key point here, I think, is not the 4-bit but the QAT — the model is trained with the objective of losing the least at 4-bit quantiZation (I am assuming it is literally about assigning numbers that quantize better). The 12B QAT model is indeed sort of mindblowing.
- verdverm 4mo ago
  I haven't dug into QAT deeply, better recovery is my understanding as well, and also that it is out of reach for most people because you have to train a model to back prop errors based on estimated error under quant. Hopefully more of the lab releases are trained under QAT so we can all benefit.
  - dofm 4mo ago
    I think they did Gemma 3 QAT models and there are QAT versions of essentially all the Gemma 4 models (including DiffusionGemma).
- novaRom 4mo ago
  Gemma 4 12B QAT is amazing - agents run very fast, and it's really very smart, at least in my agent's harness domain which is GNU software development - on par with frontiers like GPT Sol, DeepSeek, or Claude - Why to buy those expensive tokens if a local tiny model performs so well?
  - gwerbin 4mo ago
    What's your harness setup? I haven't had this kind of on-par success with any local LLM yet.
    - sosodev 4mo ago
      They’re exaggerating or have a very simple way of using these models. The Gemma 4 series, even at 31B, is nowhere near the frontier. They’re great models, but you will notice a huge difference for complex tasks. The best local agentic coding experience I’ve had so far is Qwen3.6-27B with Pi.
      - verdverm 4mo ago
        same with opencode, next on my evals.list are the fine tunes from big model traces, and then my own if ftuning looks reasonable cost/time wise I have two main tasks I want to see if I can improve, coding and doc understanding/summary
      - xienze 4mo ago
        > They’re great models, but you will notice a huge difference for complex tasks. Yes and no. I think where frontier models really blow small models away is in how thorough they are in order to infer your intentions and how best to accomplish them. So you can tell Claude "change this code to make it do X", whereas a Qwen3.6-27B or Gemma4-31B can do the same, but you have to be a lot more thorough, i.e., "change this code to make it do X, but first, let me explain the concept of X as I see it and some notes about things to avoid or pay attention to while you're doing it." So for best success with small models you really need a big toolbelt of skills and MCPs.

dofm 4mo ago

The key point here, I think, is not the 4-bit but the QAT — the model is trained with the objective of losing the least at 4-bit quantiZation (I am assuming it is literally about assigning numbers that quantize better). The 12B QAT model is indeed sort of mindblowing.

verdverm 4mo ago
I haven't dug into QAT deeply, better recovery is my understanding as well, and also that it is out of reach for most people because you have to train a model to back prop errors based on estimated error under quant. Hopefully more of the lab releases are trained under QAT so we can all benefit.
- dofm 4mo ago
  I think they did Gemma 3 QAT models and there are QAT versions of essentially all the Gemma 4 models (including DiffusionGemma).
novaRom 4mo ago
Gemma 4 12B QAT is amazing - agents run very fast, and it's really very smart, at least in my agent's harness domain which is GNU software development - on par with frontiers like GPT Sol, DeepSeek, or Claude - Why to buy those expensive tokens if a local tiny model performs so well?
- gwerbin 4mo ago
  What's your harness setup? I haven't had this kind of on-par success with any local LLM yet.
  - sosodev 4mo ago
    They’re exaggerating or have a very simple way of using these models. The Gemma 4 series, even at 31B, is nowhere near the frontier. They’re great models, but you will notice a huge difference for complex tasks. The best local agentic coding experience I’ve had so far is Qwen3.6-27B with Pi.
    - verdverm 4mo ago
      same with opencode, next on my evals.list are the fine tunes from big model traces, and then my own if ftuning looks reasonable cost/time wise I have two main tasks I want to see if I can improve, coding and doc understanding/summary
    - xienze 4mo ago
      > They’re great models, but you will notice a huge difference for complex tasks. Yes and no. I think where frontier models really blow small models away is in how thorough they are in order to infer your intentions and how best to accomplish them. So you can tell Claude "change this code to make it do X", whereas a Qwen3.6-27B or Gemma4-31B can do the same, but you have to be a lot more thorough, i.e., "change this code to make it do X, but first, let me explain the concept of X as I see it and some notes about things to avoid or pay attention to while you're doing it." So for best success with small models you really need a big toolbelt of skills and MCPs.