3 ms·
Technically it doesn't have to be since that part of the context window would have been in the KV cache and the inference provider could have thrown away the te
by red2awn 1y ago
Technically it doesn't have to be since that part of the context window would have been in the KV cache and the inference provider could have thrown away the textual input.
- Cr8 1y agopossible - but KV caches are generally _much_ bigger than the source text and can be reproduced from the source text so it wouldn't make a lot of sense to throw it out