That's not how it works. While there are some research papers looking into using chunks of kv cache while discarding others, those involve architecture changes

That's not how it works. While there are some research papers looking into using chunks of kv cache while discarding others, those involve architecture changes in the models themselves. HiCache / LMcache (used in sglang/vllm) are ways of optimising the juggling of kv cache, but IIUC it still needs to be token perfect atm.