In answer to your question: I think the underlying problem is that because of how transformer models are built. We need neural net designs that have a dedicated

In answer to your question: I think the underlying problem is that because of how transformer models are built. We need neural net designs that have a dedicated write only output layer and a read only input layer. How to use backpropagation on them: no idea. Maybe a stricter descriminator/generator training loop could lead towards that goal? Maybe we should go back to DNCs and the idea of vectorized memory layers? But the current designs always lead to the assembly problem. As long as there is no difference between control flow and data, data can always be used as control flow.