5 ms·

Is there a way to say: I am happy to pay a premium (in tokens or extra usage) to make sure that my resumed 1h+ session has all the old thinking? I understand y

by btown 6mo ago

Is there a way to say: I am happy to pay a premium (in tokens or extra usage) to make sure that my resumed 1h+ session has all the old thinking? I understand you wouldn't want this to be the default, particularly for people who have one giant running session for many topics - and I can only imagine the load involved in full cache misses at scale. But there are other use cases where this thinking is critical - for instance, a session for a large refactor or a devops/operations use case consolidating numerous issue reports and external findings over time, where the periodic thinking was actually critical to how the session evolved. For example, if N-4 was a massive dump of some relevant, some irrelevant material (say, investigating for patterns in a massive set of data, but prompted to be concise in output), then N-4's thinking might have been critical to N-2 not getting over-fixated on that dump from N-4. I'd consider it mission-critical, and pay a premium, when resuming an N some hours later to avoid pitfalls just as N-2 avoided those pitfalls. Could we have an "ultraresume" that, similar to ultrathink, would let a user indicate they want to watch Return of the (Thin)king: Extended Edition?

CjHuber 6mo ago
I think it’s crazy that they do this, especially without any notice. I would not have renewed my subscription if I knew that they started doing this. Especially in the analysis part of my work I don‘t care about the actual text output itself most of the time but try to make the model „understand“ the topic. In the first phase the actual text output itself is worthless it just serves as an indicator that the context was processed correctly and the future actual analysis work can depend on it. And they‘re… just throwing most the relevant stuff out all out without any notice when I resume my session after a few days? This is insane, Claude literally became useless to me and I didn’t even know it until now, wasting a lot of my time building up good session context. There would be nothing lost if they said „If you click yes, we will prune your old thinking making Claude faster and saving you tons of tokens“. Most people would say yes probably so why not ask them… make it an env variable (that is announced not a secretly introduced one to opt out of something new!) or at least write it in a change log if they really don’t want to allow people to use it like before, so there‘d be chance to cancel the subscription in time instead of wasting tons of time on work patterns that not longer work
- jetbalsa 6mo ago
  So to defend a litte, its a Cache, it has to go somewhere, its a save state of the model's inner workings at the time of the last message. so if it expires, it has to process the whole thing again. most people don't understand that every message the ENTIRE history of the conversion is processed again and again without that cache. That conversion might of hit several gigs worth of model weights and are you expecting them to keep that around for /all/ of your conversions you have had with it in separate sessions?
  - 3836293648 6mo ago
    No? It's not because it's a cache, it's because they're scared of letting you see the thinking trace. If you got the trace you could just send it back in full when it got evicted from the cache. This is how open weight models work.
    - eknkc 6mo ago
      I’m not familiar with the Claude API but OpenAI has an encrypted thking messages option. You get something that you can send back but it is encrypted. Not available on Anthropic?
    - reactordev 6mo ago
      They are sending it back to the cache, the part you are missing is they were charging you for it.
      - eknkc 6mo ago
        The blog post says they prune them now not to charge you. That’s the change they implemented.
        reactordev 6mo ago
        right. they were charging you for it, now they aren't because they are just dropping your conversation history.
elAhmo 6mo ago
Don't you have that by just resuming old convo? The only issue is that it didn't hit the cache so it was expensive if you resume later.
- eknkc 6mo ago
  Not at the moment apparently. They remove the thinking messages when you continue after 1 hour. That was the whole idea of that change. So the LLM gets all your messages, its responses etc but not the thinking parts, why it generated that responses. You get a lobotomised session.
  - elAhmo 6mo ago
    OK didn't know that. I also resume fairly old sessions with 100-200k of context, and I sometimes keep them active for a while (but with large breaks in between). Still on Opus 4.6 with no adaptive thinking, so didn't really notice anything worse in the past weeks, but who knows.
- tbrockman 6mo ago
  Or generate tiny filler messages every hour until you come back to it.
trinsic2 6mo ago
Why cant you just build a project document that outlines that prompt that you want to do? Or have claude save your progress in memory so you can pick it up later? Thats what I do. It seems abhorrent to expect to have a running prompt that left idle for long periods of time just so you can pick up at a moments whim...
- Terretta 6mo ago
  You know that memory goes back into a prompt as context that wasn't cached, so... that just adds work. Granted, the "memory" can be available across session, as can docs...
- try-working 6mo ago
  recursive-mode does just that: https://recursive-mode.dev/introduction https://recursive-mode.dev/introduction

by btown 6mo ago

CjHuber 6mo ago

I think it’s crazy that they do this, especially without any notice. I would not have renewed my subscription if I knew that they started doing this. Especially in the analysis part of my work I don‘t care about the actual text output itself most of the time but try to make the model „understand“ the topic. In the first phase the actual text output itself is worthless it just serves as an indicator that the context was processed correctly and the future actual analysis work can depend on it. And they‘re… just throwing most the relevant stuff out all out without any notice when I resume my session after a few days? This is insane, Claude literally became useless to me and I didn’t even know it until now, wasting a lot of my time building up good session context. There would be nothing lost if they said „If you click yes, we will prune your old thinking making Claude faster and saving you tons of tokens“. Most people would say yes probably so why not ask them… make it an env variable (that is announced not a secretly introduced one to opt out of something new!) or at least write it in a change log if they really don’t want to allow people to use it like before, so there‘d be chance to cancel the subscription in time instead of wasting tons of time on work patterns that not longer work

jetbalsa 6mo ago
So to defend a litte, its a Cache, it has to go somewhere, its a save state of the model's inner workings at the time of the last message. so if it expires, it has to process the whole thing again. most people don't understand that every message the ENTIRE history of the conversion is processed again and again without that cache. That conversion might of hit several gigs worth of model weights and are you expecting them to keep that around for /all/ of your conversions you have had with it in separate sessions?
- 3836293648 6mo ago
  No? It's not because it's a cache, it's because they're scared of letting you see the thinking trace. If you got the trace you could just send it back in full when it got evicted from the cache. This is how open weight models work.
  - eknkc 6mo ago
    I’m not familiar with the Claude API but OpenAI has an encrypted thking messages option. You get something that you can send back but it is encrypted. Not available on Anthropic?
  - reactordev 6mo ago
    They are sending it back to the cache, the part you are missing is they were charging you for it.
    - eknkc 6mo ago
      The blog post says they prune them now not to charge you. That’s the change they implemented.
      - reactordev 6mo ago
        right. they were charging you for it, now they aren't because they are just dropping your conversation history.

elAhmo 6mo ago

Don't you have that by just resuming old convo? The only issue is that it didn't hit the cache so it was expensive if you resume later.

eknkc 6mo ago
Not at the moment apparently. They remove the thinking messages when you continue after 1 hour. That was the whole idea of that change. So the LLM gets all your messages, its responses etc but not the thinking parts, why it generated that responses. You get a lobotomised session.
- elAhmo 6mo ago
  OK didn't know that. I also resume fairly old sessions with 100-200k of context, and I sometimes keep them active for a while (but with large breaks in between). Still on Opus 4.6 with no adaptive thinking, so didn't really notice anything worse in the past weeks, but who knows.
tbrockman 6mo ago
Or generate tiny filler messages every hour until you come back to it.

trinsic2 6mo ago

Why cant you just build a project document that outlines that prompt that you want to do? Or have claude save your progress in memory so you can pick it up later? Thats what I do. It seems abhorrent to expect to have a running prompt that left idle for long periods of time just so you can pick up at a moments whim...

Terretta 6mo ago
You know that memory goes back into a prompt as context that wasn't cached, so... that just adds work. Granted, the "memory" can be available across session, as can docs...
try-working 6mo ago
recursive-mode does just that: https://recursive-mode.dev/introduction https://recursive-mode.dev/introduction