It's probably not programmed to forget, but it was too expensive to implement remembering. Also probably not realted, but don't these LLMs only work with a rel

It's probably not programmed to forget, but it was too expensive to implement remembering. Also probably not realted, but don't these LLMs only work with a relatively short buffer or else they start being completely incoherent?