4 ms·
Feels like will be shut down real quick
by alukin 2mo ago
Feels like will be shut down real quick
- l1n 2mo agoWhy? Seems fine if people want to burn more cache reads.
- deleted 2mo ago[deleted]
- agluszak 2mo agoBecause it makes Anthropic earn less money
- cadamsdotcom 2mo ago"Earn"? Cache duration is arbitrary. What it actually does (if used en masse) is decrease the amount of oversubscription their infra can handle..
- janderson215 2mo agoEarn is synonymous with profit, and this hurts their profits. Introducing latency can benefit much more people and actually make usage more efficient. Reducing latency can sieze up a functioning system. Lately, I’ve been thinking about how this related to fractional banking. If you were to eliminate fractional banking introduced in the US by Hamilton, you would destroy a lot of current prosperity.
- sillysaurusx 2mo agoIt’s unknown whether our current economic model will work out long term. It mostly relies on the US being the default reserve currency for most nations. Hopefully that lasts a long time.
- munk-a 2mo ago> Because it makes Anthropic burn more money Fixed that for you.
- gogobio 2mo agoYes, this wastes cycle. They dump your cache and deallocate the VMs so that others can use it. This will result in tokens wasted and downtime for others.
- supern0va 2mo agoCan you expand on this? I can't imagine this is holding up an entire machine or materially impacting capacity. Isn't it common to have a tiered cache and to ultimately evict (or move to another tier) if there's an active inference request and no available capacity elsewhere?
- Lalabadie 2mo agoYes, however if a tool like this commoditizes a way to stay "higher" in the cache hierarchy to avoid getting dropped, then for the same total capacity, Anthropic have to become stricter about retention.
- supern0va 2mo agoIt's unfortunate that there isn't some better way to signal that there's a high chance that a turn is coming (ie, due to active sub-agents). If CC could send a ping saying "I don't need a turn, but please keep this cached at some tier because there will be a turn soon." that would probably help with cache efficiency. Instead, it seems like there's just the shotgun one-hour solution.
- Cyberdog 2mo agoAgreed, the Thermos company's lawyers will probably fire off a terse letter once they find out.