3 ms·
Qwen is a little fussy about the sampler settings, but it does run well quantized. If you were getting infinite repetition loops, try dropping the top_p a bit.
by behohippy 2y ago
Qwen is a little fussy about the sampler settings, but it does run well quantized. If you were getting infinite repetition loops, try dropping the top_p a bit. I think qwen likes lower temps too
- Eisenstein 2y agoWe are talking about dynamically quantizing KV cache, not the model weights.
- behohippy 2y agoI run the KV cache at Q8 even on that model. Is it not working well for you?