3 ms·
I've not noticed this as a specific critique of Word2Vec models, but despite their near-magic for some purposes it's well-understood that they don't capture all
by gojomo 10y ago
I've not noticed this as a specific critique of Word2Vec models, but despite their near-magic for some purposes it's well-understood that they don't capture all interesting shades of human-perceived meaning – at least not yet.
There are some interesting extensions of the basic model which include echoes of this 'tags' idea.
First, some projects have also modeled each word's vector as a sum-of-subvectors, based on the word's substrings-of-various-lengths. (Notably this is a feature of Facebook's FastText library.) This also learns useful vectors for subword features (like linguistic morphemes/roots), which then lets models often bootstrap useful vectors for new words, not included in the training corpus, based on their similarity with known words. (While those are strictly composing 'tags' of orthographic representation, it would be natural to seek a way to decompose words into varying-sized collections of more primal concepts/denotations/connotations.)
Second, modeling longer texts (like sentences) as collections-of-word-vectors gives some interesting ways to calculate similarities between texts. A first, simple way is to treat a text as some (perhaps significance-weighted) average of all its words, thus getting a single vector, in the word-space, for the text. Another is to treat the text as having an invisible floating pseudo-word that tries to help in the same word-prediction task that's used to create word-vectors. When that pseudo-word is trained-up, as if it were a rare word, treat it as a vector for the text. (That's Mikolov/Le's 'Paragraph Vector', often referred-to as 'Doc2Vec'.)
But following a practice useful from image-color-histogram comparisons, you can also treat the constituent word-vectors of text as if they were 'piles of meaning' at their individual vector coordinates, and calculate the distance between texts as the sum-of-distances/efforts to move one sentence's pile configuration to another. With text this has been called "Word Mover's Distance" but is analogous to "Earth Mover's Distance"/"Wasserstein Distance" in statistics/probability.
As this has been applied to texts – multi-word collections – I could easily see it also applied to decomposed words/concepts - multi-tag collections, per your summary of Tversky's model above. At some point, I'd also expect these models to account for words/sentences as large 'smears' in space, rather than single points, to better capture the full range of possible meanings, and then likely intended meanings, based on other relevant context.