4 ms·
It highly depends on the model and the context use. A model like command-r for instance is practically unaffected by it, but Qwen will go nuts. As well, tasks h
by Eisenstein 2y ago
It highly depends on the model and the context use. A model like command-r for instance is practically unaffected by it, but Qwen will go nuts. As well, tasks highly dependent on context like translation or evaluation will be more impacted than say, code generation or creative output.
- wkat4242 2y agoInteresting. I didn't know that. I thought it was basically 'free' space saving. Would you know how llama3.1 fares by any chance?
- behohippy 2y agoQwen is a little fussy about the sampler settings, but it does run well quantized. If you were getting infinite repetition loops, try dropping the top_p a bit. I think qwen likes lower temps too
- Eisenstein 2y agoWe are talking about dynamically quantizing KV cache, not the model weights.
- behohippy 2y agoI run the KV cache at Q8 even on that model. Is it not working well for you?