Author here. Was not expecting to see this pop up on Hacker News! This was a short write-up of a set of experiments exploring the importance of attention in tr

Author here. Was not expecting to see this pop up on Hacker News! This was a short write-up of a set of experiments exploring the importance of attention in transformers. I'm glad to see that people are still reading it and hopefully finding it interesting. Side note: the title should probably have (2021) in it, as this was posted to Arxiv in May 2021 (concurrently with a number of similar works such as MLP-Mixer).