This model has attention, just the sequence in broken into chunks of length 4096 and the attention is only applied for the chunk. Llama 2 was trained on chunks

This model has attention, just the sequence in broken into chunks of length 4096 and the attention is only applied for the chunk. Llama 2 was trained on chunks of length 4096 so this model has the same quadratic complexity for any sequence which fits within llama 2 context size.