There’s actually research being done in this space that you might find interesting: “attention sinks” https://arxiv.org/abs/2503.08908 https://arxiv.org/abs/25

There’s actually research being done in this space that you might find interesting: “attention sinks” https://arxiv.org/abs/2503.08908 https://arxiv.org/abs/2503.08908