3 ms·
Because if you don't, you trash the KV cache and things get way more expensive. We wrote about it here: https://earendil.com/posts/prompt-caching/ https://earen
by the_mitsuhiko 2mo ago
Because if you don't, you trash the KV cache and things get way more expensive. We wrote about it here: https://earendil.com/posts/prompt-caching/ https://earendil.com/posts/prompt-caching/
Encrypted reasoning content is one of the things we keep as blobs in the transcript, even though we can only send it back to the original provider that created them, as otherwise we cannot keep caches live.
- captainmuon 2mo agoRight, but my point was, I should be able to just send something like {'continue_conversation_id': 123, 'message': 'another message?'} Seems like it would be even less work for the LLM host to just check if ID 123 is still in a cache (and if not, load it from a database) than decoding my request and checking cryptographic signatures. Right now the whole conversation history with blobs functions like a really long ID.
- the_mitsuhiko 2mo agoBut then on a cache miss you need to resend the whole thing. Cached websocket connections on OpenAI work like this in a way. But once you lose the connection then you lose that. Your idea though is the core of the responses API with store where everything is stored on their side and then you append to it. For the LLM there is no difference btw. In either way you need to locate the cache and use it.