Current models go beyond the simple embedding because you start to encode groups of concepts in the context-aware part of the model (attention heads or any othe

Current models go beyond the simple embedding because you start to encode groups of concepts in the context-aware part of the model (attention heads or any other method). So it is never simply words/tokens in isolation anymore.