3 ms·

what do you mean by reset at every turn? context stays until compaction. if you remove the reasoning tokens after every turn you will be constantly blowing cach

by jolux 2mo ago

what do you mean by reset at every turn? context stays until compaction. if you remove the reasoning tokens after every turn you will be constantly blowing cache which is far worse than filling up context.

tym0 2mo ago
That's not my understanding of how most agents work. This is what a chain of request/response looks like: Your Prompt 1: Prompt Content 1 -> cache-1 LLM Response 1: <Thinking>Thinking Content 1</Thinking> Response Content 1 Your Prompt 2 (client side): prompt-1 + response-without-thinking-1 + Prompt Content 2 Your Prompt 2 (server side): cache-1 + response-without-thinking-1 + Prompt Content 2 -> cache-2 LLM Response 2: <Thinking>Thinking Content 2</Thinking> Response Content 2 Etc... So reasoning gets dropped from context and you still get cache from the accumulating requests. Edit: I've realised I was incorrect, the thinking doesn't get passed back and forth but the latent snapshot does which result in using memory just the same.
- mike_hearn 2mo ago
  Modern protocols loop back the reasoning tokens in raw textual form via an encrypted parameter. You can't see them (modulo the recent attack), but you do resubmit them.
  - tym0 2mo ago
    Yeah I've done more research and that's what I meant in the edit you replied to. But that's not the full reasoning token context, just a snapshot of the latent state at the end of it, no? Have a look at the gemini ones they're pretty small.
    - mike_hearn 2mo ago
      There was an attack that convinced models to leak the contents of the reasoning tokens, and it came back as text that matched the length of the encrypted data very well. So it's probably still tokens. Looping back latent state isn't that easy. The hidden data is the entire contents of the KV cache which can be massive. I don't think any provider is trying to loop the KV cache through the client, and neural compressions of the reasoning would be lossy / an advanced technique that is still firmly in the realm of research papers, as I understand.

by jolux 2mo ago

tym0 2mo ago

That's not my understanding of how most agents work. This is what a chain of request/response looks like: Your Prompt 1: Prompt Content 1 -> cache-1 LLM Response 1: <Thinking>Thinking Content 1</Thinking> Response Content 1 Your Prompt 2 (client side): prompt-1 + response-without-thinking-1 + Prompt Content 2 Your Prompt 2 (server side): cache-1 + response-without-thinking-1 + Prompt Content 2 -> cache-2 LLM Response 2: <Thinking>Thinking Content 2</Thinking> Response Content 2 Etc... So reasoning gets dropped from context and you still get cache from the accumulating requests. Edit: I've realised I was incorrect, the thinking doesn't get passed back and forth but the latent snapshot does which result in using memory just the same.

mike_hearn 2mo ago
Modern protocols loop back the reasoning tokens in raw textual form via an encrypted parameter. You can't see them (modulo the recent attack), but you do resubmit them.
- tym0 2mo ago
  Yeah I've done more research and that's what I meant in the edit you replied to. But that's not the full reasoning token context, just a snapshot of the latent state at the end of it, no? Have a look at the gemini ones they're pretty small.
  - mike_hearn 2mo ago
    There was an attack that convinced models to leak the contents of the reasoning tokens, and it came back as text that matched the length of the encrypted data very well. So it's probably still tokens. Looping back latent state isn't that easy. The hidden data is the entire contents of the KV cache which can be massive. I don't think any provider is trying to loop the KV cache through the client, and neural compressions of the reasoning would be lossy / an advanced technique that is still firmly in the realm of research papers, as I understand.