4 ms·

I skimmed over a couple of the papers referenced to get an idea of what optimizations LMCache is doing. * KV cache compression - compressing the bytes of the K

by alyxya 1y ago

I skimmed over a couple of the papers referenced to get an idea of what optimizations LMCache is doing. * KV cache compression - compressing the bytes of the KV cache, taking advantage of patterns in the KV cache and with dynamic levels of compression * KV cache blending - concatenating the KV caches of multiple reused prompts with minimal KV cache recomputation for use cases like RAG, where it's more performant than the standard lossless KV cache prefix optimization, and gives better results than naively concatenating the KV caches for the reused prompts These optimizations are pretty cool and different than the standard KV cache optimizations. The title saying lossless seems misleading though.

tucnak 1y ago
"Blending," or translating arbitrary substrings to prefixes, is a real curious one, & likely become a prerequisite for running dataset-scale LLM inferences at scale. See https://arxiv.org/abs/2405.16444v3 https://arxiv.org/abs/2405.16444v3 > To speed up the prefill of the long LLM inputs, one can pre-compute the KV cache of a text and re-use the KV cache when the context is reused as the prefix of another LLM input. However, the reused text chunks are not always the input prefix, which makes precomputed KV caches not directly usable since they ignore the text’s cross-attention with the preceding texts. Thus, the benefits of reusing KV caches remain largely unrealized. > This paper tackles just one challenge: when an LLM input contains multiple text chunks, how to quickly combine their precomputed KV caches in order to achieve the same generation quality as the expensive full prefill (i.e., without reusing KV cache)? [..] We present a scheme that reuses the pre-computed KV caches, regardless prefix or not, and selectively recomputes the KV values of a small subset of tokens to partially update each reused KV cache. I had recently touched on benefits of compute-in-network for KV cache management https://news.ycombinator.com/item?id=44371227 https://news.ycombinator.com/item?id=44371227 largely making arguments contra Bluefield. The CacheBlend authors note that the delay from recomputing some tokens can be hidden by pipelining it with KV loads. Note that the various systolic array/NoC architectures are well-suited for accelerating string matching tasks. A compute-in-network FPGA could therefore manage the entire process: identify viable chunks by indexing and matching of the hot substrings, prefetch the corresponding KV caches from network storage, and stitch up a new prefix before passing it to the primary inference hardware. It may as well be one of those weird cases where hard-coding the algorithm is possible in theory, but intractable in practice—because the optimal paths would be highly-dependent on topology. Nobody wants one-trick hardware. In view of Xilinx acquisition, AMD's death in the AI space appears to be greatly exaggerated!
3abiton 1y ago
I am curious about also the varying quantization of kv cache. It seems quantizing values yield better results than doing so to keys
PoignardAzur 1y ago
KV cache blending sounds like it would be super useful for Copilot-style code completion models. You could cache the contents of each file, the edits made so far, the project README, recent commits, etc, separately, and blend them dynamically depending on what the user is doing.

by alyxya 1y ago

tucnak 1y ago

"Blending," or translating arbitrary substrings to prefixes, is a real curious one, & likely become a prerequisite for running dataset-scale LLM inferences at scale. See https://arxiv.org/abs/2405.16444v3 https://arxiv.org/abs/2405.16444v3 > To speed up the prefill of the long LLM inputs, one can pre-compute the KV cache of a text and re-use the KV cache when the context is reused as the prefix of another LLM input. However, the reused text chunks are not always the input prefix, which makes precomputed KV caches not directly usable since they ignore the text’s cross-attention with the preceding texts. Thus, the benefits of reusing KV caches remain largely unrealized. > This paper tackles just one challenge: when an LLM input contains multiple text chunks, how to quickly combine their precomputed KV caches in order to achieve the same generation quality as the expensive full prefill (i.e., without reusing KV cache)? [..] We present a scheme that reuses the pre-computed KV caches, regardless prefix or not, and selectively recomputes the KV values of a small subset of tokens to partially update each reused KV cache. I had recently touched on benefits of compute-in-network for KV cache management https://news.ycombinator.com/item?id=44371227 https://news.ycombinator.com/item?id=44371227 largely making arguments contra Bluefield. The CacheBlend authors note that the delay from recomputing some tokens can be hidden by pipelining it with KV loads. Note that the various systolic array/NoC architectures are well-suited for accelerating string matching tasks. A compute-in-network FPGA could therefore manage the entire process: identify viable chunks by indexing and matching of the hot substrings, prefetch the corresponding KV caches from network storage, and stitch up a new prefix before passing it to the primary inference hardware. It may as well be one of those weird cases where hard-coding the algorithm is possible in theory, but intractable in practice—because the optimal paths would be highly-dependent on topology. Nobody wants one-trick hardware. In view of Xilinx acquisition, AMD's death in the AI space appears to be greatly exaggerated!

3abiton 1y ago

I am curious about also the varying quantization of kv cache. It seems quantizing values yield better results than doing so to keys

PoignardAzur 1y ago

KV cache blending sounds like it would be super useful for Copilot-style code completion models. You could cache the contents of each file, the edits made so far, the project README, recent commits, etc, separately, and blend them dynamically depending on what the user is doing.