3 ms·There are ways to quantize or compress KV cache down.by visarga 6mo agoThere are ways to quantize or compress KV cache down.