3 ms·
This is just making it more expensive for everyone else, right? How Claude handles its sessions is none of my business. I'm going to let them do the best they
by SwellJoe 3mo ago
This is just making it more expensive for everyone else, right?
How Claude handles its sessions is none of my business. I'm going to let them do the best they can to provide good service for everyone, and if they can't/won't, I'll switch to a provider that can.
Using these massive models is already pretty danged extravagant, I'm not going to demand to be at the front of the queue at all times, too.
- s0ck_r4w 3mo agoHow do a few extra requests with the same prefix make it more expensive for everyone else?
- sznio 3mo agoyou use more memory?
- SwellJoe 3mo agoKeeping a conversation with a very large model active requires hundreds of GB of memory. If my conversation can never be swapped out, like when I go to lunch or take an afternoon walk, that's several hundred GB not available for other users. If everybody does that, Anthropic needs even more infrastructure than the quarter trillion dollars in infra they're already using.
- s0ck_r4w 3mo agoThe idea here is that your conversation is being swapped out for no good reason. The session is still active, as the subagents continue to make progress. How is this comparable to going to lunch or taking a walk?
- SwellJoe 3mo agoIs that actually happening, though?
- devnonymous 3mo agoYes https://github.com/izeigerman/claude-thermos/blob/main/src%2Fclaude_thermos%2Fwarmer.py#L131 https://github.com/izeigerman/claude-thermos/blob/main/src%2...
- idonotknowwhy 3mo agoCC actually prompts Claude about this by default in the ~20k system prompt and instructs it to avoid 300s timeout and to be mindful of the 300s cache expiration.
- dannyw 3mo agoCC also will also block sleeps longer than eg 300s; the harness handholds the model quite a bit.
- dannyw 3mo agoBut you pay for cache writes, it's not like you're abusing anything, you're simply choosing to call a billed API for the explicitly intended purpose of writing/refreshing cache. If you keep this running for hours without doing anything, it will drain your limits and API. The use case of keeping the main thread cache warm while subagents work is very genuine and legitimate.
- deleted 3mo ago[deleted]
- kakugawa 3mo agoThe prefix cache is a resource shared by all users. This is basically a tragedy of the commons.
- fearmerchant 3mo agoIf true, I suspect Anthropic will begin suspending accounts that use it.
- dannyw 3mo agoI highly doubt that. Keeping your cache warm is a good thing, caching saves compute and electricity. Cached input is cheap for a reason, it is in everyone’s mutual interests to maximise cache hit rates.
- OccamsMirror 3mo agoIf that was true then Anthropic would have built this functionality in themselves.
- brookst 3mo agoThey did. On any long running query with large context, tell Claude “give me a status report every 4 minutes to keep the cache warm”. It’s entirely happy to oblige and usually says “good idea, that will keep costs down”
- dannyw 3mo agoCompute is also a resource shared by all users. When your prompt isn’t cached, your input needs to go through prefill; which is compute-bound. The cache is discounted for a reason. They WANT you to use it.
- searealist 3mo agoThey could always just extend the cache timeout beyond 5 minutes themselves. They don't do that because it is an expensive resource and there is a trade-off bewteen saving computation and reserving VRAM. Running a tool like this will force them to do one of the following: 1) Start charging for VRAM reservations. 2) Charge _other_ customers more. 3) Eat the cost themselves.
- devnonymous 3mo agoFrom the README: > Detect the danger window. When the main lineage goes idle and a subagent is actively running, the main prefix is at risk of expiring. So, this is not demanding to be at the front of the queue, it's just paying someone to take the place you already had in the queue, when you want to take a leak.
- SwellJoe 3mo agoA subagent running doesn't mean that much, though. I often walk away while the subagents are chewing.
- devnonymous 3mo agoThe defaults seem sensible. This isn't keep-alive-forever by default solution.
- pkulak 3mo agoEh, I assume Anthropic prices queries somewhere in line with their actual costs. If they don't, and they misalign incentives instead, that's kinda on them.
- GMoromisato 3mo agoI agree with you. And sadly, there's nothing we can do about it because it's an arms race. People will try to exploit loopholes to get more benefits and companies will try to prevent exploits--sometimes by making the service worse for normal users. In this specific case, Anthropic can avoid keeping the cache if it detects this kind of prompt (i.e., if max tokens < some number).
- engineer_22 3mo agoYou’re already paying for the subagent, why should you pay twice? (Once for subagent, and again for waiting for the subagent). That’s double billing.
- dannyw 3mo agoYou PAY for cache writes. Holding the prefill in VRAM for a certain period of time is part of what you're paying for.
- gruez 3mo agoRight, but also using the cache refreshes/extends it, which means you can pay for a write once, then have it extended for very cheap (1 token).
- spacemanspiff01 3mo agoYou dont have to repay per hour? It seems like that is a design issue - Like yes we will keep these in whatever cache you want, but you will have some sort of renewal period
- icase 3mo agohey guys dario’s here!
- SwellJoe 3mo agoThere's no need for insults.
- himata4113 3mo agoNo? You still pay for cache reads... Actually this is actually MORE expensive than just redoing prefill if your context is below 100k.
- brookst 3mo agoThis has nothing to do with queueing. Cached input tokens cost 10% of uncached. So if you’re model runs for 45 minutes, generates 300k output tokens and asks you a question, it costs 10x more if you wait 5.01 minutes to answer. Sure, you may be willing to pay 10x more (or get 10x less for your subscription). But the time limit is arbitrary and has nothing to do with other peoples’ workloads. So I think your point is a non sequitur.
- Jensson 3mo ago> But the time limit is arbitrary and has nothing to do with other peoples’ workloads No it has to do with others workloads, now you keep their cache for longer so others will get less. And no its not arbitrary, they run out of memory, if more people do this they will have the dial it down further or run out of capacity.
- brookst 3mo agoExplain to me how “every gets 5 minutes, 24/7, regardless of load” is remotely compatible with your statement? Are you imagining this a fixed MRU where duration scales with usage? Becasue that is not at all what Anthropic documents: https://platform.claude.com/docs/en/build-with-claude/prompt-caching https://platform.claude.com/docs/en/build-with-claude/prompt... You would not get more than 5 minutes if you were the only user in the world. You would not get less at their peak hours.
- searealist 3mo ago1) To be clear, it costs you 10x more for uncached input tokens _for the next call_, which are still 5x cheaper than output tokens. 2) Now imagine Anthropic or OpenAI now charge your per minute of reserved VRAM time. It would be more fair if they did. Would you still want to run a tool like this?
- brookst 3mo agoTotally agree that if the economics changed, the economics would change.