9 ms·
Sounds good. I saw that you use the FP8 version of the model. Do you also quantize the KV cache?
by Tepix 6mo ago
Sounds good. I saw that you use the FP8 version of the model. Do you also quantize the KV cache?
- sacrelege 6mo agono I don't, since there seem to be a silent degradation bug