2 ms·
Author here. Context for why I did this: There's a growing literature arguing LRU is the wrong eviction policy for agentic LLM serving, because agent sessions
by gauravapiscean 2mo ago
Author here. Context for why I did this:
There's a growing literature arguing LRU is the wrong eviction policy for agentic LLM serving, because agent sessions idle and LRU can't distinguish a paused session from a dead one. I found the argument convincing and built a simulator to exploit it. Three separate mechanisms, all lost to plain radix-leaf LRU.
The reason turned out to be more useful than the policy. When I measured — policy-independently — where recompute actually comes from on 393 real Claude Code sessions, requests arriving after a gap longer than the 5-minute provider TTL account for 17.5% of it. Requests arriving within 10 seconds account for 33.1%. The dominant waste is tight tool loops whose 88k-token working sets exceed cache capacity, not sessions idling past a TTL. That's a capacity problem, and liveness prediction can't touch it.
- Shadowmist 2mo ago> Author here. The best kind of correct.
- r_lee 2mo agojust curious, why the LLM writing even here? it's just a bit disheartening to read Claude output for such a small comment like this.. it'd be great to read your own writings even if it's not as "perfect"
- pmarreck 2mo agoOne side effect of extremely-accessible, high-quality English translation and, more importantly, English-grammar-and-idiosyncrasy-obeying AI, is that there will be more and more text that looks like this which comes from foreign, non-native-English countries. Just FYI. This will be especially true from non-English cultures where "avoiding shame" is high on the list of motivations. Consider the upside, though: A much broader range of written perspectives written in high-quality, if slightly annoying, English. I'm already seeing the benefit of this on X thanks to its autotranslation btw: I follow a few Chinese-language accounts now that I would have never been able to digest otherwise.
- HarHarVeryFunny 2mo agoWhy/when would people expect agents to be idling? I'd have thought they'd be busy (using cached prompt-prefixes) until they were finished.
- what 2mo agoMaybe waiting on a slow tool call or subagent or something?