You've hit the nail on the head: the early Word2Vec models treat everything as a word, but words are polysemous and fluid. Mutual information is useful for disc

You've hit the nail on the head: the early Word2Vec models treat everything as a word, but words are polysemous and fluid. Mutual information is useful for discovering n-grams that can be informatively mapped to the same vector space as component 1-grams. (In fact, converting adjacent words with high pointwise mutual information such as "New" and "Zealand" to "New_Zealand" is an important preprocessing step noted by Mikolov, et al). More recent vector embeddings (e.g., BERT, ELMO) are harder to conceptualize neatly because one of the layers of the Neural Network includes a token's indexical position in the document.