Most recently, there's "Transformer-XL: Language Modeling with Longer-Term Dependency" (https://openreview.net/forum?id=HJePno0cYm https://openreview.net/forum?

Most recently, there's "Transformer-XL: Language Modeling with Longer-Term Dependency" (https://openreview.net/forum?id=HJePno0cYm https://openreview.net/forum?id=HJePno0cYm), which uses a rather different approach to apply a recurrent element to the Transformer architecture and thereby attain very long context lengths.