Suppose someone asked you to complete the sentence: “After I woke up and made breakfast, I drank a glass of …” In America one might say the most likely next w

Suppose someone asked you to complete the sentence: “After I woke up and made breakfast, I drank a glass of …” In America one might say the most likely next words are “orange juice”, or “apple juice” but not “sports car” which has nothing to do with the sentence. Ultimately this is what language models do, given a sequence of data (in this case words) predict the most likely next word(s). For attention, when you read the sentence, which words stood out as more important? Probably woke up, breakfast, and glass while the words after, I, and made were less important to completing the sentence. That is, you paid more attention to the important words to understand how to complete the sentence. The “attention mechanism” in language models is a way to let the models learn which words are important in sentences and pay more attention to them too when completing sentences, just like a person would do as in the example above. Further, it turns out this attention mechanism lets the models do lots of interesting things even without other fancy model techniques. That is “attention is all you need”.