3 ms·
Compute is also a resource shared by all users. When your prompt isn’t cached, your input needs to go through prefill; which is compute-bound. The cache is dis
by dannyw 3mo ago
Compute is also a resource shared by all users. When your prompt isn’t cached, your input needs to go through prefill; which is compute-bound.
The cache is discounted for a reason. They WANT you to use it.
- searealist 3mo agoThey could always just extend the cache timeout beyond 5 minutes themselves. They don't do that because it is an expensive resource and there is a trade-off bewteen saving computation and reserving VRAM. Running a tool like this will force them to do one of the following: 1) Start charging for VRAM reservations. 2) Charge _other_ customers more. 3) Eat the cost themselves.
- dannyw 3mo agoThey charge you for cache writes already. There's a price assigned to it. Even if you're on subscription, it counts against your usage (very easy to observe with a big cache miss). Anthropic (and now OpenAI too for 5.6) prompt caching is not free.
- searealist 3mo agoHolding your cache in VRAM for 5 minutes or 3 hours have very different costs to them. "They already charge me to park my car, why can't I leave it there for a year for the same price as 1 week?"