3 ms·
This has nothing to do with queueing. Cached input tokens cost 10% of uncached. So if you’re model runs for 45 minutes, generates 300k output tokens and asks y
by brookst 3mo ago
This has nothing to do with queueing.
Cached input tokens cost 10% of uncached. So if you’re model runs for 45 minutes, generates 300k output tokens and asks you a question, it costs 10x more if you wait 5.01 minutes to answer.
Sure, you may be willing to pay 10x more (or get 10x less for your subscription). But the time limit is arbitrary and has nothing to do with other peoples’ workloads. So I think your point is a non sequitur.
- Jensson 3mo ago> But the time limit is arbitrary and has nothing to do with other peoples’ workloads No it has to do with others workloads, now you keep their cache for longer so others will get less. And no its not arbitrary, they run out of memory, if more people do this they will have the dial it down further or run out of capacity.
- brookst 3mo agoExplain to me how “every gets 5 minutes, 24/7, regardless of load” is remotely compatible with your statement? Are you imagining this a fixed MRU where duration scales with usage? Becasue that is not at all what Anthropic documents: https://platform.claude.com/docs/en/build-with-claude/prompt-caching https://platform.claude.com/docs/en/build-with-claude/prompt... You would not get more than 5 minutes if you were the only user in the world. You would not get less at their peak hours.
- searealist 3mo ago1) To be clear, it costs you 10x more for uncached input tokens _for the next call_, which are still 5x cheaper than output tokens. 2) Now imagine Anthropic or OpenAI now charge your per minute of reserved VRAM time. It would be more fair if they did. Would you still want to run a tool like this?
- brookst 3mo agoTotally agree that if the economics changed, the economics would change.
- toasty228 3mo ago> has nothing to do with other peoples’ workloads. Because ram/memory is free and not in demande at all these days?