4 ms·
I’m guessing that there’s a system prompt at the top telling the model about its reasoning budget. So when you switch reasoning effort it busts the cache.
by janalsncm 2mo ago
I’m guessing that there’s a system prompt at the top telling the model about its reasoning budget. So when you switch reasoning effort it busts the cache.
- foota 2mo agoHmmm... Why wouldn't this be handled like other end of prompt things like the current mode?
- janalsncm 2mo agoPrompt caching only works based on the prefix. Let’s call your output Y and the low reasoning prompt A and a medium reasoning prompt B. Previously you were at A+Y. Switching to medium reasoning makes it B+Y. There’s no prefix which can be cached, so the entire B+Y needs to be reprocessed.
- foota 2mo agoWhy can't effort just be at the end of the prompt? My understanding is that that is how modes and other "dynamic updates" like the time are handled.
- SoMomentary 2mo agoThat makes sense to me. The output styles work the same way.
- cellularmitosis 2mo agoMaybe switching effort routes you to a different rack of gpu’s which don’t have the cache
- janalsncm 2mo agoThe KV cache is probably offloaded to RAM after a generation is complete. Then it is pulled back into any rack in the data center that has the model you’re using.