ELI5 - the big problem for computers working with language is that a computer doesn’t know what words mean or which words are related to each other. The main u

ELI5 - the big problem for computers working with language is that a computer doesn’t know what words mean or which words are related to each other. The main useful idea from ML is just that we could learn meaning directly from the data—and we have a lot of data thanks to the internet. So there was a lot of work that went into ways to learn the meaning of every word and the relationships between words directly from text data—with some impressive successes. But in almost all cases one of the biggest problems was learning how words affect each other when they’re far apart. That was a really hard problem because if you want to know how any two words affect each other then there are a lot of pairs you need to try. If you have 10 words in a sentence then there’s about 100 pairs; and if you have 1000 words then you have about 1 million pairs. For many years it seemed silly to even try that; computers are fast, but not _that_ fast…right? But eventually hardware got powerful enough that someone decided to throw away all the cleverness and complexity-instead they just did the most obvious thing: try _every_ pair of words. When you get down to it, that’s really all that attention is: just test every pair of inputs to see how similar they are. The title of the paper “Attention is All You Need” highlights that you can get rid of all the other tricks that people had been inventing to work around this problem of relating words that are far apart and learning the right meaning from the data. You don’t need to remember earlier words, you don’t need a fixed size context window or dynamic context or pre-trained word vectors or many, many other ideas. You _just_ need Attention to learn what words mean and solve the long distance problem. Now, it didn’t _really_ solve the problem because the original transformer could only handle around 500 tokens. This is what folks mean when they talk about the “context length” or “context window” of a Transformer model. And it’s why everyone has been so surprised and impressed when the context window for GPT jumped to 2,000 (that’s 16x more memory than the original transformer), and now we see models with 30k or 100k context windows. In any case, at this point we’ve learned that the (relatively) simple idea of Transformers is actually incredibly powerful—and remarkably general-purpose. There are actually only a handful of new ideas in Transformer-based models today than in the original paper.