This seems like a definite attention optimization but I think the fundamental problem with attention is that it doesn’t handle state in a way that scales well.

This seems like a definite attention optimization but I think the fundamental problem with attention is that it doesn’t handle state in a way that scales well. Personally I think the RNN/LSTM state handling approach is going to be something we revisit when trying to advance past transformers. It handles state in a way that generalizes and scales better (it should in theory learn an attention-like mechanism anyway, and state is independent of input size). It may be harder to train, and require further improvements, but it really seems more like an engineering or cost problem than a theoretical one. But I’m only an amateur and not an expert. Maybe continued improvement on attention will approach generalized state handling in a way that efficiently trains better than improvements on more generalized stateful approaches improve training.