4 ms·
They could always just extend the cache timeout beyond 5 minutes themselves. They don't do that because it is an expensive resource and there is a trade-off bew
by searealist 2mo ago
They could always just extend the cache timeout beyond 5 minutes themselves. They don't do that because it is an expensive resource and there is a trade-off bewteen saving computation and reserving VRAM. Running a tool like this will force them to do one of the following:
1) Start charging for VRAM reservations.
2) Charge _other_ customers more.
3) Eat the cost themselves.
- dannyw 2mo agoThey charge you for cache writes already. There's a price assigned to it. Even if you're on subscription, it counts against your usage (very easy to observe with a big cache miss). Anthropic (and now OpenAI too for 5.6) prompt caching is not free.
- searealist 2mo agoHolding your cache in VRAM for 5 minutes or 3 hours have very different costs to them. "They already charge me to park my car, why can't I leave it there for a year for the same price as 1 week?"