4 ms·
Hmm. I admit that I haven't thought about this deeply, but I'm not sure that's true? It seems to me that you could extend the KV cache either backwards or forwa
by kcorbitt 3y ago
Hmm. I admit that I haven't thought about this deeply, but I'm not sure that's true? It seems to me that you could extend the KV cache either backwards or forwards equally easily.
- Siira 3y agoYou can’t. The later values depend on the earlier ones, so changing the early tokens invalidates your whole cache. This is also probably why leading with a question works better in the first place. All later processing conditions on the question in this way. BTW, in my very limited testing, GPT4 doesn’t care about the order.