In 2016 Transformers didn't exist and the state of the art for neural network based NLP was using LSTMs that had a limit of maybe 100 words at most. With new i

In 2016 Transformers didn't exist and the state of the art for neural network based NLP was using LSTMs that had a limit of maybe 100 words at most. With new implementations like xformers[1] and flash attention[2] it is unclear where the length limit is on modern transformer models. Flash Attention can currently scale up to 64,000 tokens on an A100. [1] https://github.com/facebookresearch/xformers/blob/main/HOWTO.md https://github.com/facebookresearch/xformers/blob/main/HOWTO... [2] https://github.com/HazyResearch/flash-attention https://github.com/HazyResearch/flash-attention