3 ms·
Can you explain how does it work? like how is the previous K/V cache used when you switch to another model? Source?
by thehamkercat 2mo ago
Can you explain how does it work? like how is the previous K/V cache used when you switch to another model?
Source?
- hedgehog 2mo agoSee sibling answer but essentially the effectiveness of cache is not diminished by having a separate one per model (relative to the win of doing more turns and generation with a cheaper model).
- try-working 2mo agoedit: updated the answer above to be more qualitative instead