3 ms·
Keeping a conversation with a very large model active requires hundreds of GB of memory. If my conversation can never be swapped out, like when I go to lunch or
by SwellJoe 3mo ago
Keeping a conversation with a very large model active requires hundreds of GB of memory. If my conversation can never be swapped out, like when I go to lunch or take an afternoon walk, that's several hundred GB not available for other users. If everybody does that, Anthropic needs even more infrastructure than the quarter trillion dollars in infra they're already using.
- s0ck_r4w 3mo agoThe idea here is that your conversation is being swapped out for no good reason. The session is still active, as the subagents continue to make progress. How is this comparable to going to lunch or taking a walk?
- SwellJoe 3mo agoIs that actually happening, though?
- devnonymous 3mo agoYes https://github.com/izeigerman/claude-thermos/blob/main/src%2Fclaude_thermos%2Fwarmer.py#L131 https://github.com/izeigerman/claude-thermos/blob/main/src%2...
- idonotknowwhy 3mo agoCC actually prompts Claude about this by default in the ~20k system prompt and instructs it to avoid 300s timeout and to be mindful of the 300s cache expiration.
- dannyw 3mo agoCC also will also block sleeps longer than eg 300s; the harness handholds the model quite a bit.
- dannyw 3mo agoBut you pay for cache writes, it's not like you're abusing anything, you're simply choosing to call a billed API for the explicitly intended purpose of writing/refreshing cache. If you keep this running for hours without doing anything, it will drain your limits and API. The use case of keeping the main thread cache warm while subagents work is very genuine and legitimate.
- deleted 3mo ago[deleted]