4 ms·
Conversations are always "reprocessed from scratch" on every message you send. LLMs are practically stateless and the conversation is the state, as in nothing i
by omneity 1y ago
Conversations are always "reprocessed from scratch" on every message you send. LLMs are practically stateless and the conversation is the state, as in nothing is kept in memory between two turns.
- Jabrov 1y agoNot exactly true ... KV and prompt caching is a thing
- yahoozoo 1y agoAssuming you include the same prompts in the new request that were cached in the previous ones.
- throw310822 1y agoAs far as I understand, the entire chat is the prompt. So at the each round, the previous chat up to that point could already be cached. If I'm not wrong, Claude APIs require an explicit request to cache the prompt, while OpenAI's handle this automatically.
- iamnotagenius 1y ago[dead]
- littlestymaar 1y agoI don't understand how you are downvoted…
- littlestymaar 1y ago> LLMs are practically stateless This isn't true of any practical implementation: for a particular conversation, KV Cache is the state. (Indeed there's no state across conversations, but that's irrelevant to the discussion). You can drop it after each response, but doing so increase the amount of token you need to process by a lot in multi-turn conversations. And my point was that storing the KV cache for the duration of the conversation isn't possible if you switch between multiple providers in a single conversation.