> Attention mechanism inference costs rise with the square of the inference window. This is incorrect. It is correct for the original Transformer, but OpenAI i

> Attention mechanism inference costs rise with the square of the inference window. This is incorrect. It is correct for the original Transformer, but OpenAI isn't using the original Transformer since GPT-3. Sparse Transformer scales O(N sqrt(N)). https://openai.com/research/sparse-transformer https://openai.com/research/sparse-transformer