5 ms·

While cool, quantization to FP4 is practically never lossless in actual use. A lot of providers are advertising high TPS on Kimi and GLM, but the models are fun

While cool, quantization to FP4 is practically never lossless in actual use. A lot of providers are advertising high TPS on Kimi and GLM, but the models are functionally lobotomized and no longer close to frontier quality. Would love to see this not be true.

google234123 3mo ago
First thing I noticed as well
tw1984 3mo ago
from memory, it is like 96-98% of the accuracy.
- lgessler 3mo ago
  Accuracy isn't a meaningful metric here without reference to a specific task.
  - flawn 3mo ago
    Additionally, I'd imagine quantization to have more side-effects than just slightly lower performance (on whatever task). You are basically removing information, and that information could be by chance what the model needs to fulfill it exactly the way you'd want to do - although it's still fully capable. I am not sure if this is really different from "lower performance" but open to hear your opinions.
- EduardoBautista 3mo ago
  And that 2%-4% makes all the difference.
  - fpaf 3mo ago
    Yes, it's like saying "we took off a big chunk of his brain but look! He can still breathe autonomously, swallow food and walk almost straight, which is like 95% of what he did before!"
unrvl22 3mo ago
MI355X can perform FP6 operations with the same speed as their FP4 (unique to AMD) - people should be making MXFP6 quants which would be pretty much lossless, and much closer to FP4 performance than FP8
- Hugsun 3mo ago
  That can only be true if the workload is compute bound, not memory bandwidth bound.
zozbot234 3mo ago
Kimi uses INT4 as its native format, there's no such thing as "better than 4-bit precision" for that model. This is in contrast with GLM for which 16-bit precision is native and 8-bit is in common use.
- hassaanr 3mo ago
  You’re right, but this poses a separate issue as the providers then do FP4 PTQ, which is quite lossy. Reduces the model size and optimizes for Blackwells at the (imo severe) cost of performance.
minraws 3mo ago
Doesn't Nvidia with their NVFP4 claim that it's lossless? I haven't tested enough models Nvidia has converted to NVFP4 besides GLM 5.2 but it seemed fine to me. My own luck has been hit or miss with it.
- HDThoreaun 3mo ago
  Certainly not lossless. Whether the loss matters depends on the range of values being quantized. When there are outliers that are massively higher than their neighbors the precision of those neighbors gets wrecked(or the outlier gets clipped), so it's important to utilize strategies that decrease the maximum value or increase the minimum. I suspect some models put more effort into that and therefore are more effective when quantized.
- suprjami 3mo ago
  Absolutely not lossless: https://www.reddit.com/r/LocalLLaMA/comments/1twz9ur/cyankiwi_awq_4bit_2605_update_nvfp4_fp8_dynamic/ https://www.reddit.com/r/LocalLLaMA/comments/1twz9ur/cyankiw...

google234123 3mo ago

First thing I noticed as well

tw1984 3mo ago

from memory, it is like 96-98% of the accuracy.

lgessler 3mo ago
Accuracy isn't a meaningful metric here without reference to a specific task.
- flawn 3mo ago
  Additionally, I'd imagine quantization to have more side-effects than just slightly lower performance (on whatever task). You are basically removing information, and that information could be by chance what the model needs to fulfill it exactly the way you'd want to do - although it's still fully capable. I am not sure if this is really different from "lower performance" but open to hear your opinions.
EduardoBautista 3mo ago
And that 2%-4% makes all the difference.
- fpaf 3mo ago
  Yes, it's like saying "we took off a big chunk of his brain but look! He can still breathe autonomously, swallow food and walk almost straight, which is like 95% of what he did before!"

unrvl22 3mo ago

MI355X can perform FP6 operations with the same speed as their FP4 (unique to AMD) - people should be making MXFP6 quants which would be pretty much lossless, and much closer to FP4 performance than FP8

Hugsun 3mo ago
That can only be true if the workload is compute bound, not memory bandwidth bound.

zozbot234 3mo ago

Kimi uses INT4 as its native format, there's no such thing as "better than 4-bit precision" for that model. This is in contrast with GLM for which 16-bit precision is native and 8-bit is in common use.

hassaanr 3mo ago
You’re right, but this poses a separate issue as the providers then do FP4 PTQ, which is quite lossy. Reduces the model size and optimizes for Blackwells at the (imo severe) cost of performance.

minraws 3mo ago

Doesn't Nvidia with their NVFP4 claim that it's lossless? I haven't tested enough models Nvidia has converted to NVFP4 besides GLM 5.2 but it seemed fine to me. My own luck has been hit or miss with it.

HDThoreaun 3mo ago
Certainly not lossless. Whether the loss matters depends on the range of values being quantized. When there are outliers that are massively higher than their neighbors the precision of those neighbors gets wrecked(or the outlier gets clipped), so it's important to utilize strategies that decrease the maximum value or increase the minimum. I suspect some models put more effort into that and therefore are more effective when quantized.
suprjami 3mo ago
Absolutely not lossless: https://www.reddit.com/r/LocalLLaMA/comments/1twz9ur/cyankiwi_awq_4bit_2605_update_nvfp4_fp8_dynamic/ https://www.reddit.com/r/LocalLLaMA/comments/1twz9ur/cyankiw...