3 ms·
We are talking about dynamically quantizing KV cache, not the model weights.
by Eisenstein 2y ago
We are talking about dynamically quantizing KV cache, not the model weights.
- behohippy 2y agoI run the KV cache at Q8 even on that model. Is it not working well for you?