2 ms·
I've seen a few articles from providers talking about KV cache quantisation, but it's not something they explicitly point out like they do with weights. So you
by scrlk 4mo ago
I've seen a few articles from providers talking about KV cache quantisation, but it's not something they explicitly point out like they do with weights.
So you could end up paying more for unquantised weights, only to get silently hit with a quantised KV cache...