4 ms·
How is it possible to do non-prefix KV cache? I was under the impression that the V for one token potentially depends on the V of all previous ones.
by dist-epoch 1y ago
How is it possible to do non-prefix KV cache? I was under the impression that the V for one token potentially depends on the V of all previous ones.
- da-x 1y agoYes, there's KV cache 'Blending' see [1]. Future versions of LMCache are aiming to support this. [1] CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion- https://arxiv.org/abs/2405.16444 https://arxiv.org/abs/2405.16444