5 ms·
Feels like will be shut down real quick
by alukin 4mo ago
Feels like will be shut down real quick
- l1n 4mo agoWhy? Seems fine if people want to burn more cache reads.
- deleted 4mo ago[deleted]
- agluszak 4mo agoBecause it makes Anthropic earn less money
- cadamsdotcom 4mo ago"Earn"? Cache duration is arbitrary. What it actually does (if used en masse) is decrease the amount of oversubscription their infra can handle..
- janderson215 4mo agoEarn is synonymous with profit, and this hurts their profits. Introducing latency can benefit much more people and actually make usage more efficient. Reducing latency can sieze up a functioning system. Lately, I’ve been thinking about how this related to fractional banking. If you were to eliminate fractional banking introduced in the US by Hamilton, you would destroy a lot of current prosperity.
- sillysaurusx 4mo agoIt’s unknown whether our current economic model will work out long term. It mostly relies on the US being the default reserve currency for most nations. Hopefully that lasts a long time.
- munk-a 4mo ago> Because it makes Anthropic burn more money Fixed that for you.
- gogobio 4mo agoYes, this wastes cycle. They dump your cache and deallocate the VMs so that others can use it. This will result in tokens wasted and downtime for others.
- supern0va 4mo agoCan you expand on this? I can't imagine this is holding up an entire machine or materially impacting capacity. Isn't it common to have a tiered cache and to ultimately evict (or move to another tier) if there's an active inference request and no available capacity elsewhere?
- Lalabadie 4mo agoYes, however if a tool like this commoditizes a way to stay "higher" in the cache hierarchy to avoid getting dropped, then for the same total capacity, Anthropic have to become stricter about retention.
- supern0va 4mo agoIt's unfortunate that there isn't some better way to signal that there's a high chance that a turn is coming (ie, due to active sub-agents). If CC could send a ping saying "I don't need a turn, but please keep this cached at some tier because there will be a turn soon." that would probably help with cache efficiency. Instead, it seems like there's just the shotgun one-hour solution.
- Cyberdog 4mo agoAgreed, the Thermos company's lawyers will probably fire off a terse letter once they find out.