This effect is well known but not well documented so far, so great job here. It's actually even more significant than it's possible to benchmark easily (though

This effect is well known but not well documented so far, so great job here. It's actually even more significant than it's possible to benchmark easily (though I'm glad this paper has done so.) Truly useful LLM applications live at the boundaries of what the model can do. That is, attending to some aspect of the context that might be several logical "hops" away from the actual question or task. I suspect that the context rot problem gets much worse for these more complex tasks... in fact, exponentially so for each logical "hop" which is required to answer successfully. Each hop compounds the "attention difficulty" which is increased by long/distracting contexts.