3 ms·

I am curious about also the varying quantization of kv cache. It seems quantizing values yield better results than doing so to keys