Long Term and Short Term "memory" is a well known concept on NLP, the predecessor to Transforms were Long-Short-Term Mememory networks aka LSTMs. And LSTMs are

Long Term and Short Term "memory" is a well known concept on NLP, the predecessor to Transforms were Long-Short-Term Mememory networks aka LSTMs. And LSTMs are just RNNs that attempt to solve the vanishing gradient problem. The whole strength of Transformers is that they are capable of processing the entire text corpus in parallel, hence they can "hold" long range dependencies better.