5 ms·

Yes. It’s perfectly reasonable to expect the user to know the intricacies of the caching strategy of their llm. Totally reasonable expectation.

coldtea 6mo ago
It's not like they have a poweful all-knowing oracle that can explain it to them at their dispos... oh, wait!
- esafak 6mo ago
  They have to know that this could bite them and to ask the question first.
  - nixpulvis 6mo ago
    I do think having some insight into the current state of the cache and a realistic estimate for prompt token use is something we should demand.
    - switchbak 6mo ago
      If there was an affordance on the TUI that made this visible and encouraged users to learn more - that would go a long way.
jghn 6mo ago
To some extent I'd say it is indeed reasonable. I had observed the effect for a while: if I walked away from a session I noticed that my next prompt would chew up a bunch of context. And that led me to do some digging, at which point I discovered their prompt caching. So while I'd agree with your sarcasm that expecting users to be experts of the system is a big ask, where I disagree with you is that users should be curious and actively attempting to understand how it works around them. Given that the tooling changes often, this is an endless job.
- abustamam 6mo ago
  > users should be curious and actively attempting to understand how it works Have you ever talked with users? > this is an endless job Indeed. If we spend all our time learning what changed with all our tooling when it changes without proper documentation then we spend all our working lives keeping up instead of doing our actual jobs.
  - Octoth0rpe 6mo ago
    There are general users of the average SaaS, and there are claude code users. There's no doubt in my mind that our expectations should be somewhat higher for CC users re: memory. I'm personally not completely convinced that cache eviction should be part of their thought process while using CC, but it's not _that_ much of a stretch.
    - abustamam 6mo ago
      Personally I've never thought about cache eviction as it pertains to CC. It's just not something that I ever needed to think about. Maybe I'm just not a power user but I just use the product the way I want to and it just works.
    - troupo 6mo ago
      Anthropic literally advertises long sessions, 1M context, high reasoning etc. And then their vibe-coders tell us that we are to blame for using the product exactly as advertised: https://x.com/lydiahallie/status/2039800718371307603 https://x.com/lydiahallie/status/2039800718371307603 while silently changing how the product works. Please stop defending hapless innocent corporations.
      - jghn 6mo ago
        This oversells how obfuscated it is. I'm far from a power user, and the opposite of a vibe coder. Yet I noticed the effect on my own just from general usage. If I can do it, anyone can do it.

coldtea 6mo ago

It's not like they have a poweful all-knowing oracle that can explain it to them at their dispos... oh, wait!

esafak 6mo ago
They have to know that this could bite them and to ask the question first.
- nixpulvis 6mo ago
  I do think having some insight into the current state of the cache and a realistic estimate for prompt token use is something we should demand.
  - switchbak 6mo ago
    If there was an affordance on the TUI that made this visible and encouraged users to learn more - that would go a long way.

jghn 6mo ago

To some extent I'd say it is indeed reasonable. I had observed the effect for a while: if I walked away from a session I noticed that my next prompt would chew up a bunch of context. And that led me to do some digging, at which point I discovered their prompt caching. So while I'd agree with your sarcasm that expecting users to be experts of the system is a big ask, where I disagree with you is that users should be curious and actively attempting to understand how it works around them. Given that the tooling changes often, this is an endless job.

abustamam 6mo ago
> users should be curious and actively attempting to understand how it works Have you ever talked with users? > this is an endless job Indeed. If we spend all our time learning what changed with all our tooling when it changes without proper documentation then we spend all our working lives keeping up instead of doing our actual jobs.
- Octoth0rpe 6mo ago
  There are general users of the average SaaS, and there are claude code users. There's no doubt in my mind that our expectations should be somewhat higher for CC users re: memory. I'm personally not completely convinced that cache eviction should be part of their thought process while using CC, but it's not _that_ much of a stretch.
  - abustamam 6mo ago
    Personally I've never thought about cache eviction as it pertains to CC. It's just not something that I ever needed to think about. Maybe I'm just not a power user but I just use the product the way I want to and it just works.
  - troupo 6mo ago
    Anthropic literally advertises long sessions, 1M context, high reasoning etc. And then their vibe-coders tell us that we are to blame for using the product exactly as advertised: https://x.com/lydiahallie/status/2039800718371307603 https://x.com/lydiahallie/status/2039800718371307603 while silently changing how the product works. Please stop defending hapless innocent corporations.
    - jghn 6mo ago
      This oversells how obfuscated it is. I'm far from a power user, and the opposite of a vibe coder. Yet I noticed the effect on my own just from general usage. If I can do it, anyone can do it.