It's not about not fitting in context window. LLMs also can "forget" the things from their early context window that they do not restate later. It's also a form

It's not about not fitting in context window. LLMs also can "forget" the things from their early context window that they do not restate later. It's also a form of decluttering. You can't (and shouldn't) remember (pay attention to) old stuff with the same priority as new, more relevant stuff.