It may be misleading to treat noncontextual word embeddings (e.g. GloVe, word2vec) and contextual word embeddings (BERT, ELMO, etc) as equivalent; there's a sub

It may be misleading to treat noncontextual word embeddings (e.g. GloVe, word2vec) and contextual word embeddings (BERT, ELMO, etc) as equivalent; there's a substantial contextual difference whether you're talking about a vector representing "boy" as a generic string or concept, equal for all mentions of "boy", or if you're talking about a vector representing a specific mention of "boy" in a specific context only, with no grounds to assume that it will be equal or even similar to other mentions of "boy". The former embeddings [try to] capture generic meaning of words, the latter [try to] capture the more specific meaning of particular utterances.