To address how computers establish context, usually the tokens are sent through an embedding layer (basically turning a word into a dense vector). Word embeddin

To address how computers establish context, usually the tokens are sent through an embedding layer (basically turning a word into a dense vector). Word embeddings, surprisingly, reflect the real-world semantic relationships behind words (for instance, king - man + woman = queen). It also happens that the algorithms for creating word embeddings group words with similar meanings/semantics together (for example, king and queen would be closer together in n-dimensional space). Overall, embeddings allow a computer to look at a language as more than a bunch of interconnected tokens and actually incorporate the semantics of each word into its predictions. Of course, this isn't perfect, which is why we need attention mechanisms and various other methods to help a computer try to understand context long-term.