5 ms·
Lossless LLM 3x Throughput Increase by LMCache
- lihanc111 1y agoOur team has built this open source project, LMCache, to reduce repetitive computation in LLM inference and make systems serve more people (3x more throughput in chat applications) and it has been used in IBM's open source LLM inference stack. In LLM serving, the input is computed into intermediate states called KV cache to further provide answers. These data are relatively large (~1-2GB for long context) and are often evicted when GPU memory is not enough. In these cases, when users ask a follow up question, the software needs to recompute for the same KV Cache. LMCache is designed to combat that by efficiently offloading and loading these KV cache to and from DRAM and disk. Ask us anything!
- dist-epoch 1y agoHow is it possible to do non-prefix KV cache? I was under the impression that the V for one token potentially depends on the V of all previous ones.
- da-x 1y agoYes, there's KV cache 'Blending' see [1]. Future versions of LMCache are aiming to support this. [1] CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion- https://arxiv.org/abs/2405.16444 https://arxiv.org/abs/2405.16444
- pama 1y agoIs your aim targetting the inference at scale or specialized/new/simpler inference pipelines? Sglang and vllm have disaggregated prefix and decoding serving (eg https://docs.vllm.ai/examples/online_serving/disaggregated_serving.html https://docs.vllm.ai/examples/online_serving/disaggregated_s... or https://github.com/sgl-project/sglang/issues/3554 https://github.com/sgl-project/sglang/issues/3554 and https://github.com/sgl-project/sglang/issues/4655 https://github.com/sgl-project/sglang/issues/4655) — could your solution enable a model-agnostic cache store/server or is that orthogonal to what you are trying to achieve?
- nativeit 1y agoHas it been used in IBM's inference stack, or used with IBM's inference stack? In other words, has this been merged into IBM's own repositories, or has someone just tested it using them?
- lihanc111 1y agoIt is in IBM's llm-d open source stack
- behnamoh 1y ago> Our team So this is something that might in the future turning to a commercial product? something like Langchain and thousands of open source projects that started as "open source" but then ended up implementing proprietary features for a cost.
- Tokumei-no-hito 1y agoi don't see anything wrong with that approach, do you?
- behnamoh 1y agoGive it time and you'll come to my conclusion.
- 0xjunhao 1y agoHi, I had a quick question. Would it be correct to say the following? 1. For long inputs and short outputs, the inference can be arbitrarily number of times faster, as it avoids repeated KV computation. 2. Conversely, for short inputs and long outputs, it might be slightly slower, since loading and storing the KV cache are on the critical path of the execution.
- lihanc111 1y agoIt is almost true for both. Although for the second case you can just skip storing in these cases where there is little improvement.
- iLoveOncall 1y agoIs this any different than prompt caching?
- smcleod 1y agoHave you considered integrating it with the likes of llama.cpp?
- m3kw9 1y agoHow would it work if a user wants to do 1 of n tries?
- kcorbitt 1y agoLooks cool! With vLLM v1, prefix caching is enabled by default and seems quite performant. Is the advantage of LMCache the fact that you can offload to CPU and disk as well? How much is throughput/latency affected if you need to pull a large KV cache from disk/cpu instead of GPU RAM? Also, how realistic would it be to share the KV cache across vllm nodes within a data center? It would be really nice to be able to freely distribute requests to a pool of vLLM workers without worrying about prefix-aware routing, but maybe that isn't the right approach because moving the KV cache around would be too slow?
- guywhocodes 1y agoThis is exactly what llm-d is
- ekianjo 1y agowasn't this already implemented in llama.cpp?
- sgammon 1y agoHey LMCache team! Saw you guys at OSS N.A. but wasn’t able to set aside time to say hello. We’d love to chat about collaborating. Is there an email we can reach out to?
- lihanc111 1y agoPlease send to contact@lmcache.ai
- refulgentis 1y agoWord to the wise: "Lossless 3x Throughput Increase" == "Cache all inputs and output across everyone, in RAM and on disk, and if you assume the next request is covered by cache, its 3x faster!" I'm more surprised it's only advertised as 3x under those conditions: my llama.cpp wrapper does the same -- caching in RAM while running locally seems fine to me -- and when input is cached, TTFT is ~instantaneous, modulo any add'l prompt you add. I supposed it creates a little more distance, in that, instead of infinity times faster for latency, we measure throughput, and then our speedup can be adjusted as desired by adjusting output length, and thus we can pick a more reasonable-sounding metric like 3x. (though, the GitHub README still frames it in terms of latency / TTFT)
- varispeed 1y agoSometimes I think the entire engineering profession collectively underwent a lobotomy. Techniques like caching partial computation results to avoid repeating expensive work were so basic a few decades ago that no one would have bothered to dignify them with a paper, let alone brand them with a fancy acronym and announce them like the second coming of Turing. Now we get breathless blog posts and community calls over the mind-blowing discovery that storing KV caches of repeated text speeds things up. Next we'll get a paper on using hash tables to look things up faster. Meanwhile, actual difficult problems in large-scale distributed inference and model interpretability get hand-waved so we can posture about reinventing memoisation. Tech never fails to take the obvious, put a bow on it, and sell it back to us as groundbreaking.
- vlovich123 1y agoPartial caching as a concept doesn’t matter. The hard part is figuring out how to make it work for cross attention which sets up a data dependency for every entry on every preceding entry. So prefix caching of KV cache is brain dead easy. Computing a KV cache for random bits of text and then combining unrelated text in a way that makes the LLM still work coherently and correctly? That to me seems much harder. It seems to me like you’re easily hand waving away a hard problem in a different part of the stack you’re less familiar with.
- varispeed 1y agoLet’s be honest: it’s fundamentally about analysing memory access patterns, spotting reuse opportunities, and orchestrating data flows. That’s classic systems engineering. Useful, yes. Rocket science, no. The real joke is how the profession has sunk so low that anything beyond a trivial for-loop becomes a grounds for whitepapers, corporate branding, and breathless conference talks. In the past, we’d have quietly shipped this and moved on. Frankly, I’m surprised they haven’t patented it yet.
- vlovich123 1y agoCaching and reuse broadly yes. Getting cross attention to work mathematically correctly by stitching the pre computed KV cache for snippets of text is not that unless you’ve redefined what classical systems engineering is. Again, the novelty is in getting cross attention to work correctly despite the fact that you’re stitching together arbitrary caches together. It’s akin to taking snippets of compressed portions of random compressed files and reconstructing a new correct plain text. That’s obviously not possible but clearly this has been accomplished with the KV cache for arbitrary models (ie not trained for it) despite the KV cache working like decompression where all the preceding bytes have to be computed correctly for the subsequent token to be correct.
- nativeit 1y agoIt seems odd to me that so many of these projects are being launched by people who have only just discovered and/or joined HN. I'm worried this is just becoming LinkedIn for AI opportunists.
- parpfish 1y agoI’ve got a side project that I may (someday) do a show HN with. However, I’d probably make a new account for that because the project is connected to my real name/portfolio and I don’t want that connected with my pseudonymous comments here
- nativeit 1y agoI considered that, but then why would anyone obfuscate this really very reasonable scenario by choosing another ostensibly pseudonymous username?
- deleted 1y ago[deleted]
- fsmv 1y ago[deleted]
- parpfish 1y agoI imagine that this is a common problem and it could be another cool “unlockable” on HN, like the downvotes at 500 karma. Once you get X karma or account age >Y years, you can make one anonymous submissions each quarter that comes from an non-user but still get some sort of “verified” badge that proves it comes from a legit user.
- refulgentis 1y agoYou nailed it IMHO. I quit my job at Google 2 years ago to do LLM stuff, was looking forward to having HN around, but discussions re: LLMs here are a minefield. Why? Everyone knows at least a little, and everyone has a strong opinion on it given the impact of it. People sharing stuff sell it way high, and as with any new thing where people are selling, there's a lot of skeptics. Then, throw in human bias towards disliking what seems like snark / complaining, so stuff with substance gets downvotes. SNR ratio is continually decreasing. Let's dig into why this one is weird: My work inferences using either 3P provider, which do caching, or llama.cpp, in which I do caching. (basically, picture it as there's a super expensive step that you can skip by keeping Map<input string, gpu state>) So I log into HN and see this and say to myself: 3x! throughput increase? This is either really clever or salesmanship, no way an optimization like that has been sitting around on the groud. So I read the GitHub, see it's just "write everyones inputs and outputs to disk, you can then use them to cobble together what the GPU state would be for an incoming request!", and write a mostly-polite comment below flagging "hey, this means writing everything to disk" Then I start replying to you...but then I throw away the comment, because I'm inviting drive-by downvotes. I.e. the minefield describe up top, and if you look like you're being mean, you'll eat downvotes, especially on a weekend. And to your average reader, maybe I just don't understand vLLM, and am taking it out in good hackers just pushing code. Then, when I go back, I immediately see a comment from someone who does use vLLM noting it already does caching. Sigh.
- wg0 1y agoSeems like snake oil to me. I mean lacks clear explanation of how exactly it works if at all.
- ahmedhawas123 1y agoLike this a lot and thanks for making it open source. Does this support ollama today? I only saw vLLM
- jbentley1 1y agoIs this the same as the prompt caching that other API's (Anthropc, OpenAI, etc) have had, just open source and for vLLM?
- alyxya 1y agoI skimmed over a couple of the papers referenced to get an idea of what optimizations LMCache is doing. * KV cache compression - compressing the bytes of the KV cache, taking advantage of patterns in the KV cache and with dynamic levels of compression * KV cache blending - concatenating the KV caches of multiple reused prompts with minimal KV cache recomputation for use cases like RAG, where it's more performant than the standard lossless KV cache prefix optimization, and gives better results than naively concatenating the KV caches for the reused prompts These optimizations are pretty cool and different than the standard KV cache optimizations. The title saying lossless seems misleading though.
- tucnak 1y ago"Blending," or translating arbitrary substrings to prefixes, is a real curious one, & likely become a prerequisite for running dataset-scale LLM inferences at scale. See https://arxiv.org/abs/2405.16444v3 https://arxiv.org/abs/2405.16444v3 > To speed up the prefill of the long LLM inputs, one can pre-compute the KV cache of a text and re-use the KV cache when the context is reused as the prefix of another LLM input. However, the reused text chunks are not always the input prefix, which makes precomputed KV caches not directly usable since they ignore the text’s cross-attention with the preceding texts. Thus, the benefits of reusing KV caches remain largely unrealized. > This paper tackles just one challenge: when an LLM input contains multiple text chunks, how to quickly combine their precomputed KV caches in order to achieve the same generation quality as the expensive full prefill (i.e., without reusing KV cache)? [..] We present a scheme that reuses the pre-computed KV caches, regardless prefix or not, and selectively recomputes the KV values of a small subset of tokens to partially update each reused KV cache. I had recently touched on benefits of compute-in-network for KV cache management https://news.ycombinator.com/item?id=44371227 https://news.ycombinator.com/item?id=44371227 largely making arguments contra Bluefield. The CacheBlend authors note that the delay from recomputing some tokens can be hidden by pipelining it with KV loads. Note that the various systolic array/NoC architectures are well-suited for accelerating string matching tasks. A compute-in-network FPGA could therefore manage the entire process: identify viable chunks by indexing and matching of the hot substrings, prefetch the corresponding KV caches from network storage, and stitch up a new prefix before passing it to the primary inference hardware. It may as well be one of those weird cases where hard-coding the algorithm is possible in theory, but intractable in practice—because the optimal paths would be highly-dependent on topology. Nobody wants one-trick hardware. In view of Xilinx acquisition, AMD's death in the AI space appears to be greatly exaggerated!
- tom910 1y agoWhere can I find more detailed explanations about how it works? A simple key/value solution based on the hash of the prompt will not work because almost every request will have a unique hash. How can I solve this problem and maintain quality?
- hasanar1f 1y agoIs LMCache entirely lossless? Cuz, the kv cache streamig in the Cachegen paper was not lossless. Or is there any way to control the loss in LMCache?