Seems related to “infinite context” by adding attention sinks. This seems to be the same mechanism at play. https://arxiv.org/abs/2309.17453.pdf https://arxiv.

Seems related to “infinite context” by adding attention sinks. This seems to be the same mechanism at play. https://arxiv.org/abs/2309.17453.pdf https://arxiv.org/abs/2309.17453.pdf