3 ms·

each layer is made up of various weights, the weights are adjusted to quant it. a pure q8 will have all the weights as q8, or a q4 the same. but some are kep

each layer is made up of various weights, the weights are adjusted to quant it. a pure q8 will have all the weights as q8, or a q4 the same. but some are kept as f32, etc. here's an example of q3_k_xl - https://huggingface.co/unsloth/Kimi-K2-Thinking-GGUF/tree/main/UD-Q3_K_XL?show_file_info=UD-Q3_K_XL%2FKimi-K2-Thinking-UD-Q3_K_XL-00007-of-00010.gguf https://huggingface.co/unsloth/Kimi-K2-Thinking-GGUF/tree/ma... we can see certain weights are f32, q8, q5, q3, etc. They used mxfp4 in some weights and mxfp4 doesn't seem to place nicely in quants so that's why they are retiring it. read their publication again and it should make more sense.

jychang 8mo ago
I am aware of all that. They literally never say “they used mxfp4 in some weights”. What you’re claiming they said doesn’t exist. This isn’t a postmortem, it’s PR fluff without actually addressing the issue.
- segmondy 8mo ago
  It's right there https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks I looked at the weights before. It's not PR fluff, they made it clear by showing how it really affected various tensors terribly. "MXFP4 is much worse on many tensors - attn_gate, attn_q, ssm_beta, ssm_alpha using MXFP4 is not a good idea, and rather Q4_K is better - also MXFP4 uses 4.25 bits per weight, whilst Q4_K uses 4.5 bits per weight. It's better to use Q4_K than MXFP4 when choosing between them." The Q4 quants had a mixture of mxfp4 leading to worse outcomes.
  - jychang 8mo ago
    Nope. Where do they say something along the lines of "we had MXFP4 tensors in our previous upload" or "that's why we re-uploaded new versions"? This is a famous non-apology non-explanation of what actually happened. "They made it clear by showing how it really affected various tensors terribly"? Where do they even say they had ever previously uploaded any quant with MXFP4?

jychang 8mo ago

I am aware of all that. They literally never say “they used mxfp4 in some weights”. What you’re claiming they said doesn’t exist. This isn’t a postmortem, it’s PR fluff without actually addressing the issue.

segmondy 8mo ago
It's right there https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks I looked at the weights before. It's not PR fluff, they made it clear by showing how it really affected various tensors terribly. "MXFP4 is much worse on many tensors - attn_gate, attn_q, ssm_beta, ssm_alpha using MXFP4 is not a good idea, and rather Q4_K is better - also MXFP4 uses 4.25 bits per weight, whilst Q4_K uses 4.5 bits per weight. It's better to use Q4_K than MXFP4 when choosing between them." The Q4 quants had a mixture of mxfp4 leading to worse outcomes.
- jychang 8mo ago
  Nope. Where do they say something along the lines of "we had MXFP4 tensors in our previous upload" or "that's why we re-uploaded new versions"? This is a famous non-apology non-explanation of what actually happened. "They made it clear by showing how it really affected various tensors terribly"? Where do they even say they had ever previously uploaded any quant with MXFP4?