4 ms·

Agents push the full conversation history into context every turn Why? Maybe this is a dumb question, but why wouldn't an agent "keep the conversation going",

Agents push the full conversation history into context every turn Why? Maybe this is a dumb question, but why wouldn't an agent "keep the conversation going", like I do when interacting with an LLM through a web page? (I understand how it's impractical for long-running tasks where the agent has to wait days for the next input, but assume that's not the majority of use cases)

BoredPositron 5mo ago
The "web page" does the same you just don't see it.
sosodev 5mo ago
I’m not sure I understand your question. Every interaction you have with a model in a web page does the same thing in the backend. It feeds the whole conversation history, perhaps with a bit of processing, into the model so it can process the next generation. Filling the context window is how these models retain coherence.
isbvhodnvemrwvn 5mo ago
LLMs are stateless, to predict next tokens they need the history. When you write your own agents you will be very selective and might trim context and heavily segment different tasks, but generic ones don't do that (at best they spawn subjects to handle smaller tasks)
- lxgr 5mo ago
  That said, the KV cache is very much not stateless, so internally inference APIs will be highly incentivized to route requests to instances with as much a shared prefix cached as possible.
- rkagerer 5mo ago
  Thanks. If I ran it local, presumably I could keep the state cached forever. Can you "reserve" resources from a frontier provider to guarantee your state stays "hot"? (Analogous to reserving a whole VM instead of a slice)
  - gpugreg 5mo ago
    For Anthropic, 5 minute caching costs 1.25x base input price and 1 hour costs 2x base input price. https://platform.claude.com/docs/en/about-claude/pricing#prompt-caching https://platform.claude.com/docs/en/about-claude/pricing#pro... For OpenAI, it seems like you can't prolong the caching duration for money. Duration is longer during off-peak hours for in-memory caching and up to 24 hours for extended prompt caching. https://developers.openai.com/api/docs/guides/prompt-caching https://developers.openai.com/api/docs/guides/prompt-caching For DeepSeek, caching duration of at least 12 hours (and likely longer) have been observed. Cache writes are free. https://zhuanlan.zhihu.com/p/2035737726952194774 https://zhuanlan.zhihu.com/p/2035737726952194774
eknkc 5mo ago
BTW, the openai responses api has a store parameter and a thread id input. Makes it possible to send a thread id and append a new message, ask for completion. So it feels like keeping the conversation going. Technically it does retrieve the entire history and reevaulate it since the LLM is stateless. Just more ergonomic for the developer. And prompt caching helps cut the costs down when a conversation drags on.
- drewnick 5mo ago
  Wow, this is refreshing DX compared to iterating all messages like we did back in '24.
  - ghrl 5mo ago
    I would disagree. Having all the messages locally and sending them with the request means you can switch inference providers or even models mid-conversation. It also means that the provider doesn't store the entire context, which often contains massive parts of proprietary codebases, secrets and PII and instead the agent harness manages all that. While a simple `continue thread` API field might seem more convenient, the cost is still determined by the input token count and cache rate, so it just abstracts this crucial implementation detail away.

BoredPositron 5mo ago

The "web page" does the same you just don't see it.

sosodev 5mo ago

I’m not sure I understand your question. Every interaction you have with a model in a web page does the same thing in the backend. It feeds the whole conversation history, perhaps with a bit of processing, into the model so it can process the next generation. Filling the context window is how these models retain coherence.

isbvhodnvemrwvn 5mo ago

LLMs are stateless, to predict next tokens they need the history. When you write your own agents you will be very selective and might trim context and heavily segment different tasks, but generic ones don't do that (at best they spawn subjects to handle smaller tasks)

lxgr 5mo ago
That said, the KV cache is very much not stateless, so internally inference APIs will be highly incentivized to route requests to instances with as much a shared prefix cached as possible.
rkagerer 5mo ago
Thanks. If I ran it local, presumably I could keep the state cached forever. Can you "reserve" resources from a frontier provider to guarantee your state stays "hot"? (Analogous to reserving a whole VM instead of a slice)
- gpugreg 5mo ago
  For Anthropic, 5 minute caching costs 1.25x base input price and 1 hour costs 2x base input price. https://platform.claude.com/docs/en/about-claude/pricing#prompt-caching https://platform.claude.com/docs/en/about-claude/pricing#pro... For OpenAI, it seems like you can't prolong the caching duration for money. Duration is longer during off-peak hours for in-memory caching and up to 24 hours for extended prompt caching. https://developers.openai.com/api/docs/guides/prompt-caching https://developers.openai.com/api/docs/guides/prompt-caching For DeepSeek, caching duration of at least 12 hours (and likely longer) have been observed. Cache writes are free. https://zhuanlan.zhihu.com/p/2035737726952194774 https://zhuanlan.zhihu.com/p/2035737726952194774

eknkc 5mo ago

BTW, the openai responses api has a store parameter and a thread id input. Makes it possible to send a thread id and append a new message, ask for completion. So it feels like keeping the conversation going. Technically it does retrieve the entire history and reevaulate it since the LLM is stateless. Just more ergonomic for the developer. And prompt caching helps cut the costs down when a conversation drags on.

drewnick 5mo ago
Wow, this is refreshing DX compared to iterating all messages like we did back in '24.
- ghrl 5mo ago
  I would disagree. Having all the messages locally and sending them with the request means you can switch inference providers or even models mid-conversation. It also means that the provider doesn't store the entire context, which often contains massive parts of proprietary codebases, secrets and PII and instead the agent harness manages all that. While a simple `continue thread` API field might seem more convenient, the cost is still determined by the input token count and cache rate, so it just abstracts this crucial implementation detail away.