I see a lot of people misunderstanding what this is about. What this allows is incrementally updating the attention cache. It does not allow the model to see be

I see a lot of people misunderstanding what this is about. What this allows is incrementally updating the attention cache. It does not allow the model to see beyond its usual attention window. As they explain in a README, only the tokens that fit into the usual window are considered -- so if you ask a question about a long book, it will only consider the last pages. But it can still be useful. Imagine this use case, where you have a chat conversation between Assistant and User. Assume that the inputs to get the next assistant response are just the past conversation turns (cut off to fit context window). So for turn 1 the input is: User: (user turn 1) For turn 2 the input is: User: (user turn 1) Assistant: (assistant turn 1) User: (user turn 2) Etc. Now, what this allows you to do is reuse the attention computed from the previous turns (since the prefix is the same). In practice, people often have a system prompt before the conversation history, which (as far a I can tell) makes this technique not applicable (the input prefix will change as soon as the conversation history is long enough that we need to start dropping the oldest turns, otherwise the system prompt would get ignored). In such case, what you could do is to cache at least the system prompt. This is also possible with https://github.com/OpenNMT/CTranslate2/blob/2203ad5c8baf878a2d08e73095421e2ba033c89c/examples/llama2/chat.py#L8 https://github.com/OpenNMT/CTranslate2/blob/2203ad5c8baf878a...