3 ms·
How does it deal with prompt caching? Do you have any resource for this, I'd like to check it out. I've heard changing models messes up caching and costs more.
by danilovilhena 4mo ago
How does it deal with prompt caching? Do you have any resource for this, I'd like to check it out.
I've heard changing models messes up caching and costs more.
- verdverm 4mo agoIf you switch models, you have no cache. There is no way to change this as they depend on sending the tokens through the weights and saving the computation results. Different model, different tokens, different weights, different calculation.
- FrancescoMassa 4mo agoActually yes because context is kept on kV cache
- verdverm 4mo agoyou cannot transfer KV cache from one model to another
- FrancescoMassa 4mo agoYou transfer context from the kv cache of model A to the kv cache of model B
- verdverm 4mo agoyou cannot do this with direct byte copy, you have to reprocess all input to rebuild the cache in model B, it's exactly like waiting hours to send a new message to an existing session and your cache has expired tl;dr - you will not get the 90% discount the first time you change models and send a message this is why I am saying you cannot "transfer" kv cache, it's just like changing models in your favorite harness and that requires reprocessing the input from scratch to build a kv cache for the model
- FrancescoMassa 4mo agoit works exactly like that...that's why you're seeing the notice con claude code when switching models
- FrancescoMassa 4mo agoFor sure that’s absolutely true because you’re switching context from a kV cache to another one. We introduced 2 algorythms for solving this : 1) sticky : switches model only when convenient 2) smartsqueeze : fast advanced compression for reingesting context