3 ms·
Very cool. For the INT4 QAT model, what is the recommended precision for the activations and for the key and values stored in KV cache?
by cgdl 1y ago
Very cool. For the INT4 QAT model, what is the recommended precision for the activations and for the key and values stored in KV cache?
- hnuser123456 1y agoFor keys, you probably want to use at least q5 or q6, for values q4 is fine