Also in that article he says "In fact, I’d go as far as to say that The concept of attention is the most interesting recent architectural innovation in ne

Also in that article he says "In fact, I’d go as far as to say that The concept of attention is the most interesting recent architectural innovation in neural networks." when the initial attention paper was less than a year old, and two years before the transformer paper.