4 ms·
Every turn of a conversation with an LLM is getting the whole conversation. Caching complicates the picture, but not by a huge amount. That's why a short questi
by SwellJoe 4mo ago
Every turn of a conversation with an LLM is getting the whole conversation. Caching complicates the picture, but not by a huge amount. That's why a short question at the end of a long conversation chews tokens faster than it would in a fresh session.
So, a conversation that's ongoing with one model then switching to another would presumably send the whole conversation and the new question. Which defeats the purpose of splitting traffic...so, you're not wrong to question how this actually improves things for anything other than short sessions, which you could choose your own model for if it's a small problem.
- nok22kon 4mo agosending the whole conversation to a cheap model could still be cheaper than sending just the latest message to the expensive one you could even take this into account automatically to help decide
- SwellJoe 4mo agoDeepSeek is seemingly novel in its caching capability. In most of my usage of it, I'm seeing something like 80% cached tokens, which makes using DeepSeek cheaper than damned near any other model anywhere near its size. I don't think even my self-hosted models are cheaper once electricity and air conditioning is accounted for (it's summer in Texas, pumping a few hundred watts of heat into the room for hours at a time for local LLM usage is sub-optimal). So, I think if you're using a model that has extremely effective caching (of which there is only one I know of) as your expensive model, you almost certainly won't come out ahead by switching back and forth. Cached tokens in DeepSeek are damned near cheap as free. But, for Claude or GPT where tokens are extremely expensive and caching barely puts a dent in it, yeah, I guess there is math that could work out in favor of this idea.