"Blending," or translating arbitrary substrings to prefixes, is a real curious one, & likely become a prerequisite for running dataset-scale LLM inferences at s

"Blending," or translating arbitrary substrings to prefixes, is a real curious one, & likely become a prerequisite for running dataset-scale LLM inferences at scale. See https://arxiv.org/abs/2405.16444v3 https://arxiv.org/abs/2405.16444v3 > To speed up the prefill of the long LLM inputs, one can pre-compute the KV cache of a text and re-use the KV cache when the context is reused as the prefix of another LLM input. However, the reused text chunks are not always the input prefix, which makes precomputed KV caches not directly usable since they ignore the text’s cross-attention with the preceding texts. Thus, the benefits of reusing KV caches remain largely unrealized. > This paper tackles just one challenge: when an LLM input contains multiple text chunks, how to quickly combine their precomputed KV caches in order to achieve the same generation quality as the expensive full prefill (i.e., without reusing KV cache)? [..] We present a scheme that reuses the pre-computed KV caches, regardless prefix or not, and selectively recomputes the KV values of a small subset of tokens to partially update each reused KV cache. I had recently touched on benefits of compute-in-network for KV cache management https://news.ycombinator.com/item?id=44371227 https://news.ycombinator.com/item?id=44371227 largely making arguments contra Bluefield. The CacheBlend authors note that the delay from recomputing some tokens can be hidden by pipelining it with KV loads. Note that the various systolic array/NoC architectures are well-suited for accelerating string matching tasks. A compute-in-network FPGA could therefore manage the entire process: identify viable chunks by indexing and matching of the hot substrings, prefetch the corresponding KV caches from network storage, and stitch up a new prefix before passing it to the primary inference hardware. It may as well be one of those weird cases where hard-coding the algorithm is possible in theory, but intractable in practice—because the optimal paths would be highly-dependent on topology. Nobody wants one-trick hardware. In view of Xilinx acquisition, AMD's death in the AI space appears to be greatly exaggerated!