3 ms·

How do a few extra requests with the same prefix make it more expensive for everyone else?

sznio 2mo ago
you use more memory?
SwellJoe 2mo ago
Keeping a conversation with a very large model active requires hundreds of GB of memory. If my conversation can never be swapped out, like when I go to lunch or take an afternoon walk, that's several hundred GB not available for other users. If everybody does that, Anthropic needs even more infrastructure than the quarter trillion dollars in infra they're already using.
- s0ck_r4w 2mo ago
  The idea here is that your conversation is being swapped out for no good reason. The session is still active, as the subagents continue to make progress. How is this comparable to going to lunch or taking a walk?
  - SwellJoe 2mo ago
    Is that actually happening, though?
    - devnonymous 2mo ago
      Yes https://github.com/izeigerman/claude-thermos/blob/main/src%2Fclaude_thermos%2Fwarmer.py#L131 https://github.com/izeigerman/claude-thermos/blob/main/src%2...
      - idonotknowwhy 2mo ago
        CC actually prompts Claude about this by default in the ~20k system prompt and instructs it to avoid 300s timeout and to be mindful of the 300s cache expiration.
        dannyw 2mo ago
        CC also will also block sleeps longer than eg 300s; the harness handholds the model quite a bit.
- dannyw 2mo ago
  But you pay for cache writes, it's not like you're abusing anything, you're simply choosing to call a billed API for the explicitly intended purpose of writing/refreshing cache. If you keep this running for hours without doing anything, it will drain your limits and API. The use case of keeping the main thread cache warm while subagents work is very genuine and legitimate.
  - deleted 2mo ago
    [deleted]
kakugawa 2mo ago
The prefix cache is a resource shared by all users. This is basically a tragedy of the commons.
- fearmerchant 2mo ago
  If true, I suspect Anthropic will begin suspending accounts that use it.
  - dannyw 2mo ago
    I highly doubt that. Keeping your cache warm is a good thing, caching saves compute and electricity. Cached input is cheap for a reason, it is in everyone’s mutual interests to maximise cache hit rates.
    - OccamsMirror 2mo ago
      If that was true then Anthropic would have built this functionality in themselves.
      - brookst 2mo ago
        They did. On any long running query with large context, tell Claude “give me a status report every 4 minutes to keep the cache warm”. It’s entirely happy to oblige and usually says “good idea, that will keep costs down”
- dannyw 2mo ago
  Compute is also a resource shared by all users. When your prompt isn’t cached, your input needs to go through prefill; which is compute-bound. The cache is discounted for a reason. They WANT you to use it.
  - searealist 2mo ago
    They could always just extend the cache timeout beyond 5 minutes themselves. They don't do that because it is an expensive resource and there is a trade-off bewteen saving computation and reserving VRAM. Running a tool like this will force them to do one of the following: 1) Start charging for VRAM reservations. 2) Charge _other_ customers more. 3) Eat the cost themselves.

sznio 2mo ago

you use more memory?

Keeping a conversation with a very large model active requires hundreds of GB of memory. If my conversation can never be swapped out, like when I go to lunch or take an afternoon walk, that's several hundred GB not available for other users. If everybody does that, Anthropic needs even more infrastructure than the quarter trillion dollars in infra they're already using.

s0ck_r4w 2mo ago
The idea here is that your conversation is being swapped out for no good reason. The session is still active, as the subagents continue to make progress. How is this comparable to going to lunch or taking a walk?
- SwellJoe 2mo ago
  Is that actually happening, though?
  - devnonymous 2mo ago
    Yes https://github.com/izeigerman/claude-thermos/blob/main/src%2Fclaude_thermos%2Fwarmer.py#L131 https://github.com/izeigerman/claude-thermos/blob/main/src%2...
    - idonotknowwhy 2mo ago
      CC actually prompts Claude about this by default in the ~20k system prompt and instructs it to avoid 300s timeout and to be mindful of the 300s cache expiration.
      - dannyw 2mo ago
        CC also will also block sleeps longer than eg 300s; the harness handholds the model quite a bit.
dannyw 2mo ago
But you pay for cache writes, it's not like you're abusing anything, you're simply choosing to call a billed API for the explicitly intended purpose of writing/refreshing cache. If you keep this running for hours without doing anything, it will drain your limits and API. The use case of keeping the main thread cache warm while subagents work is very genuine and legitimate.
- deleted 2mo ago
  [deleted]

kakugawa 2mo ago

The prefix cache is a resource shared by all users. This is basically a tragedy of the commons.

fearmerchant 2mo ago
If true, I suspect Anthropic will begin suspending accounts that use it.
- dannyw 2mo ago
  I highly doubt that. Keeping your cache warm is a good thing, caching saves compute and electricity. Cached input is cheap for a reason, it is in everyone’s mutual interests to maximise cache hit rates.
  - OccamsMirror 2mo ago
    If that was true then Anthropic would have built this functionality in themselves.
    - brookst 2mo ago
      They did. On any long running query with large context, tell Claude “give me a status report every 4 minutes to keep the cache warm”. It’s entirely happy to oblige and usually says “good idea, that will keep costs down”
dannyw 2mo ago
Compute is also a resource shared by all users. When your prompt isn’t cached, your input needs to go through prefill; which is compute-bound. The cache is discounted for a reason. They WANT you to use it.
- searealist 2mo ago
  They could always just extend the cache timeout beyond 5 minutes themselves. They don't do that because it is an expensive resource and there is a trade-off bewteen saving computation and reserving VRAM. Running a tool like this will force them to do one of the following: 1) Start charging for VRAM reservations. 2) Charge _other_ customers more. 3) Eat the cost themselves.