3 ms·
Am I missing something? As far as I can see this patch does nothing except add new options that replicate the functionality of the existing --cache-type-k and -
by therealsmith 1y ago
Am I missing something? As far as I can see this patch does nothing except add new options that replicate the functionality of the existing --cache-type-k and --cache-type-v options.
Using `--flash-attn --cache-type-k q8_0 --cache-type-v q8_0` is a very well known optimization to save VRAM.
And it's also very well known that the keys are more sensitive to quantization than values. E.g. https://arxiv.org/abs/2502.15075 https://arxiv.org/abs/2502.15075
- Aurornis 1y ago> Using `--flash-attn --cache-type-k q8_0 --cache-type-v q8_0` I think you meant ‘--cache-type-v q4_0’ I would also like an explanation for what’s different in this patch compared to the standard command line arguments.
- deleted 1y ago[deleted]
- deleted 1y ago[deleted]