Today I ran some perplexity benchmarks comparing F16 and Q8_0 for the K/V, I used Qwen 2.5 Coder 7b as I've heard people say things to the effect of Qwen being

Today I ran some perplexity benchmarks comparing F16 and Q8_0 for the K/V, I used Qwen 2.5 Coder 7b as I've heard people say things to the effect of Qwen being more sensitive to quantisation than some other models. Well, it turns out there's barely any increase in perplexity at all - an increase of just 0.0043. Added to the post: https://smcleod.net/2024/12/bringing-k/v-context-quantisation-to-ollama/#perplexity-measurements https://smcleod.net/2024/12/bringing-k/v-context-quantisatio...