4 ms·
Context caching guide
- mritchie712 3y agoI really want this (in OpenAI API). Does google not care about the reputation it's getting with this shit: > We'll be launching context caching soon, along with technical documentation and SDK support. it's not live and god knows when it will be
- nextworddev 3y agoThey have adopted the strategy of mainly releasing to & iterating with the large enterprise customers, because they (rightly) realized that GA'ing to every developer makes no monetary sense, unless they are trying to learn something from that launch / need to do so for competitive reasons.
- mritchie712 3y agofair point
- refulgentis 3y agoPresumably, this would be a competitive reason, no? It would further the cost savings they GA'd with Gemini Flash, and it's a differentiator from every other provider.
- aiauthoritydev 3y agoThis actually is smarter strategy IMO. Throwing the hat in the ring forces your competitors think about it and make them work too. It also gives you first mover advantage making people think you did it first. A lot of AI tools make more sense for their enterprise customers rather than people like us. People like us are good only for hype and not making money.
- flipbrad 3y agoMight help show prior art to defeat software patents too.
- hidelooktropic 3y agoIsn't this what the Assistants API is meant for? Honestly not sure, I haven't used it before but their documentation seems to suggest you can set up the assistant to already have this context, then just send API commands to it without said context.
- swyx 3y agobut they charge u for the full context every time, at least for now. latency would suggest that no caching is happening
- sebzim4500 3y agoMakes sense. It isn't in the list of suggested usecases, but I wonder if this can be used to speed up tree of thoughts or similar prompt/search techniques. It could also speed up restricted generation, e.g. when you force the model to output valid json.
- resource_waste 3y ago[flagged]
- _pdp_ 3y agoWe envisioned a system like this over a year ago, but since we lacked technical capabilities in this area, we couldn't solve it. To me, it is clear that a lot of information is being transmitted needlessly. There is no need to send an entire book every single time an interaction occurs. Instead, you can start from a checkpoint, remap from memory, and continue from there. Perhaps none of the current LLM service providers have an architecture that allows this, but it seems like something that should be possible and is likely to emerge in the near future.
- gradys 3y agoThe size of the cached internal state of the network processing the book is much larger than the size of the book. The resource that is preserved with caching is the compute required to recreate that state.
- dfgtyu65r 3y agoSure, but a direct forwards pass of the book would surely require more compute than simply loading and setting the hidden state? The second doesn't require any matrix operations, it's just setting some values.
- nancarrow 3y ago"some" is doing a lot of lifting. # of tokens * # of layers * head dimension * # of heads * 2 (K+V vectors) * 4-16bits (depending on quantization)
- gradys 3y agoI don’t know of any public details on how they implement Context Caching, but that is presumably exactly what they are doing. Just caching the text would be a minimal savings.
- rfoo 3y ago> it's just setting some values But it may very well be slower than just recompute it. At least for ordinary MHA and even GQA. So, either a model arch woodoo significantly reducing kv cache size (while keeping roughly the same compute cost), or some really careful implementation moving kv cache of upcoming requests to devices in background [0]. [0] My back of envelop calc shows that even then it still does not make sense for, say, Llama 3 70B on H100s. Time to stare at TPU spec harder trying to make sense of it I guess.
- sshumaker 3y agoThis is a pretty standard technique if you're running the models yourself. e.g. ChatGPT almost certainly does this. There's even work that is more sophisticated in this domain that allows 'template' style partial caching: https://arxiv.org/abs/2311.04934 https://arxiv.org/abs/2311.04934
- lolpanda 3y agoi think llama.cpp has context caching with "--prompt-cache" but it will result in a very large cache file. i guess it's also very expensive for any inference api provider to support caching as they have to persist the file and load/unload it each time.
- _cs2017_ 3y agoHow is this different from KV caching? (For reference, basic KV caching is explained here: https://www.omrimallis.com/posts/techniques-for-kv-cache-optimization/ https://www.omrimallis.com/posts/techniques-for-kv-cache-opt....)
- luckyt 3y agoThere are several issues that make the KV cache as-is unsuitable for caching across requests. First, it requires the cached tokens to be in the exact same position in the sentence, this means it's mainly only useful for autoregressive generation where the prefix is always the same. Second, it is extremely big, so without some sort of compression, the cost to store it between requests and the time required to transfer the data to the GPU will outweigh any compute savings.
- waldrews 3y agoThe pricing is still such that you can't routinely use a customized Gemini with your own long fixed pre-filled context + a variable short query. If the one-time compute cost of the caching could be effectively amortized over many queries, this pattern would replace many fine-tuning and RAG cases with something more predictable and controllable. It's not as simple as that because the large cache needs to be loaded into GPU memory every time, but optimizations must be feasible if the usage rate is large enough to keep the cache alive in a dedicated machine. Previous discussion: https://news.ycombinator.com/item?id=40034972#40036309 https://news.ycombinator.com/item?id=40034972#40036309
- navaed01 3y agoI’m really glad they released context caching- the are legitimate use cases where a single or multiple users in an organization are all using a long prompt with examples in it. Which impacts cost and speed