2 ms·
How can the smaller Unsloth GGUF quant can beat the original google quant? (ref: unsloth/gemma-4-31B-it-qat-GGUF)
by zkmon 5mo ago
How can the smaller Unsloth GGUF quant can beat the original google quant? (ref: unsloth/gemma-4-31B-it-qat-GGUF)
- SubiculumCode 4mo agoI may be wrong, but this is what I figured out. Google provided these quantize-ready models, but they do not come pre-quantized. However, to produce their benchmarks, they quantized their model using the standard quantization approach. Unsloth has an advanced quantization method that performs better than the standard quantization, so the evals are better for unsloth quants.