It’s not a mystery how they work! The paper has been out for years now. Moreover, all of the research is focused on extending memory through other means since V

It’s not a mystery how they work! The paper has been out for years now. Moreover, all of the research is focused on extending memory through other means since VRAM usage is proportional to the SQUARE in-text memory of the model. So the way forward isn’t a larger context window, but instead an external memory apparatus of some means.