> Maybe I'm missing that its usage started before that. Afaik it wasn't used prior to it. While attention (and residual connections) were used with recurrent

> Maybe I'm missing that its usage started before that. Afaik it wasn't used prior to it. While attention (and residual connections) were used with recurrent / auto regressive models, the paper showed that an encoder-decoder (non auto regressive and auto-regressive) architecture with just attention (and residuals) is sufficient to achieve great results given that you provide positional embeddings. So prior to transformers it was RNN + Attention (+ Residuals), but the paper claimed that Attention (+Residuals+Pos Embeddings) is all you need.