I agree that it would be much better if a model could move the knowledge to the weights, but I don't think it's necessary. Models like RNN have practically infi

I agree that it would be much better if a model could move the knowledge to the weights, but I don't think it's necessary. Models like RNN have practically infinite lossy "context window", and I think it would be much more reasonable to find an RNN architecture that would scale like a transformer than what you propose, while still achieving our goal. An alternative analogy you can think of is that the frozen weights in the model are our DNA, while the state or "context window" of the model is our state.