I agree it's very useful to look at RNNs. Even more because attention mechanisms were already getting popular in LSTM/GRU models before the transformers paper.

I agree it's very useful to look at RNNs. Even more because attention mechanisms were already getting popular in LSTM/GRU models before the transformers paper. One of the main insights of the transformer papers is that you can do with just an attention attention and you do not need the recurrent part of the network anymore (hence the title "Attention is all you need").