3 ms·

I'm reading the New Michael Lewis book right now, on kahneman and tversky, and there's a point where the notion of encoding similarity as distances is first ver

I'm reading the New Michael Lewis book right now, on kahneman and tversky, and there's a point where the notion of encoding similarity as distances is first very popular, but then torn apart in the sixties. Essentially, it comes down to statements like 'this woman is like Queen Elizabeth' being different from 'Queen Elizabeth is like this woman.' human perception of similarity is assymetric. Tversky's suggestion is that we essentially think if things as a collection of tags, and A is like B if A's tags are mostly a subset of B's tags. But if B has more tags (because it's a specific thing that we know a lot about, for example) then B's tags won't mostly be a subset of A's tags, creating the asymmetry. This directly attacks the kind of similarity that word2vec relies on, and I'm wondering if there are critiques along these lines on the literature.

stared 10y ago
word2vec is a naive technique, which relies only on the colocation of words. Yes, it misses many things - word order, grammar (e.g. negations), words with many distinct meanings (lead (Pb) vs (to) lead), synonyms vs antonyms etc. What's interesting is that it works at all. My blog post on that: http://p.migdal.pl/2017/01/06/king-man-woman-queen-why.html http://p.migdal.pl/2017/01/06/king-man-woman-queen-why.html
gojomo 10y ago
I've not noticed this as a specific critique of Word2Vec models, but despite their near-magic for some purposes it's well-understood that they don't capture all interesting shades of human-perceived meaning – at least not yet. There are some interesting extensions of the basic model which include echoes of this 'tags' idea. First, some projects have also modeled each word's vector as a sum-of-subvectors, based on the word's substrings-of-various-lengths. (Notably this is a feature of Facebook's FastText library.) This also learns useful vectors for subword features (like linguistic morphemes/roots), which then lets models often bootstrap useful vectors for new words, not included in the training corpus, based on their similarity with known words. (While those are strictly composing 'tags' of orthographic representation, it would be natural to seek a way to decompose words into varying-sized collections of more primal concepts/denotations/connotations.) Second, modeling longer texts (like sentences) as collections-of-word-vectors gives some interesting ways to calculate similarities between texts. A first, simple way is to treat a text as some (perhaps significance-weighted) average of all its words, thus getting a single vector, in the word-space, for the text. Another is to treat the text as having an invisible floating pseudo-word that tries to help in the same word-prediction task that's used to create word-vectors. When that pseudo-word is trained-up, as if it were a rare word, treat it as a vector for the text. (That's Mikolov/Le's 'Paragraph Vector', often referred-to as 'Doc2Vec'.) But following a practice useful from image-color-histogram comparisons, you can also treat the constituent word-vectors of text as if they were 'piles of meaning' at their individual vector coordinates, and calculate the distance between texts as the sum-of-distances/efforts to move one sentence's pile configuration to another. With text this has been called "Word Mover's Distance" but is analogous to "Earth Mover's Distance"/"Wasserstein Distance" in statistics/probability. As this has been applied to texts – multi-word collections – I could easily see it also applied to decomposed words/concepts - multi-tag collections, per your summary of Tversky's model above. At some point, I'd also expect these models to account for words/sentences as large 'smears' in space, rather than single points, to better capture the full range of possible meanings, and then likely intended meanings, based on other relevant context.

stared 10y ago

word2vec is a naive technique, which relies only on the colocation of words. Yes, it misses many things - word order, grammar (e.g. negations), words with many distinct meanings (lead (Pb) vs (to) lead), synonyms vs antonyms etc. What's interesting is that it works at all. My blog post on that: http://p.migdal.pl/2017/01/06/king-man-woman-queen-why.html http://p.migdal.pl/2017/01/06/king-man-woman-queen-why.html

gojomo 10y ago

I've not noticed this as a specific critique of Word2Vec models, but despite their near-magic for some purposes it's well-understood that they don't capture all interesting shades of human-perceived meaning – at least not yet. There are some interesting extensions of the basic model which include echoes of this 'tags' idea. First, some projects have also modeled each word's vector as a sum-of-subvectors, based on the word's substrings-of-various-lengths. (Notably this is a feature of Facebook's FastText library.) This also learns useful vectors for subword features (like linguistic morphemes/roots), which then lets models often bootstrap useful vectors for new words, not included in the training corpus, based on their similarity with known words. (While those are strictly composing 'tags' of orthographic representation, it would be natural to seek a way to decompose words into varying-sized collections of more primal concepts/denotations/connotations.) Second, modeling longer texts (like sentences) as collections-of-word-vectors gives some interesting ways to calculate similarities between texts. A first, simple way is to treat a text as some (perhaps significance-weighted) average of all its words, thus getting a single vector, in the word-space, for the text. Another is to treat the text as having an invisible floating pseudo-word that tries to help in the same word-prediction task that's used to create word-vectors. When that pseudo-word is trained-up, as if it were a rare word, treat it as a vector for the text. (That's Mikolov/Le's 'Paragraph Vector', often referred-to as 'Doc2Vec'.) But following a practice useful from image-color-histogram comparisons, you can also treat the constituent word-vectors of text as if they were 'piles of meaning' at their individual vector coordinates, and calculate the distance between texts as the sum-of-distances/efforts to move one sentence's pile configuration to another. With text this has been called "Word Mover's Distance" but is analogous to "Earth Mover's Distance"/"Wasserstein Distance" in statistics/probability. As this has been applied to texts – multi-word collections – I could easily see it also applied to decomposed words/concepts - multi-tag collections, per your summary of Tversky's model above. At some point, I'd also expect these models to account for words/sentences as large 'smears' in space, rather than single points, to better capture the full range of possible meanings, and then likely intended meanings, based on other relevant context.