2 ms·

The word2vec inspired tricks like king-man+woman only work if the embedding is trained with synonym/antonym triplets to give them the semantic locality that all

The word2vec inspired tricks like king-man+woman only work if the embedding is trained with synonym/antonym triplets to give them the semantic locality that allows that kind of vector math. This isn't always done, even some word2vec re-implementations skip this step completely. Also, not all embeddings are word embeddings.

mikewarot 3y ago
My understanding was that Word2Vec[1] was trained on Wikipedia and other such texts, not artificially constructed things like the triplets you suggest. There's an inherent structure present in human languages that enable the "magic" of embeddings to work, as far as I can tell. [1] https://code.google.com/archive/p/word2vec/ https://code.google.com/archive/p/word2vec/
- itronitron 3y ago
  Has there been any rigorous evaluation of word2vec calculating 'king-man+woman=queen' associations? I only recall the author providing some cherry-picked examples from their results, which I suppose makes it a seminal AI paper.
  - kevindamm 3y ago
    The original paper included source, and that has their test data and results -- it gets ~77% accuracy on about 20k example word analogies (with 99.7% coverage), and 78% accuracy with phrases with 77% coverage (and a much smaller data set, 3,218 examples). You can see the test sets here: https://github.com/tmikolov/word2vec/blob/master/questions-words.txt https://github.com/tmikolov/word2vec/blob/master/questions-w... https://github.com/tmikolov/word2vec/blob/master/questions-phrases.txt https://github.com/tmikolov/word2vec/blob/master/questions-p... and you could see how much better LLMs do on the same 20k examples.
  - sp332 3y ago
    A lot more info about how it worked here https://blog.esciencecenter.nl/king-man-woman-king-9a7fd2935a85 https://blog.esciencecenter.nl/king-man-woman-king-9a7fd2935... and an interactive demo https://dash.gallery/dash-word-arithmetic/ https://dash.gallery/dash-word-arithmetic/ (blog post about the demo https://medium.com/plotly/understanding-word-embedding-arithmetic-why-theres-no-single-answer-to-king-man-woman-cd2760e2cb7f https://medium.com/plotly/understanding-word-embedding-arith...)
  - 0xdeadbeefbabe 3y ago
    If a paper attacks plus and minus it ought to cite this one.
- kevindamm 3y ago
  Ah, yes I may have mis-remembered or misunderstood when looking at the training data and model definition, it would have been about a decade ago now. Or perhaps I was thinking of an unrelated experiment to use chosen analogies during training but yeah that isn't part of the original paper. The training process from the original Mikolov et al. paper only uses the analogy examples (questions-words.txt and questions-phrases.txt) to measure accuracy after training: https://github.com/tmikolov/word2vec/blob/20c129af10659f7c50e86e3be406df663beff438/demo-train-big-model-v1.sh#L98 https://github.com/tmikolov/word2vec/blob/20c129af10659f7c50...
- _t89y 3y ago
  There is an inherent structure in language. Embeddings do not and will not capture it. It's why they do not work. Their ability to form grammatical sentences with high accuracy is part of the illusion that you have been understood.

mikewarot 3y ago

My understanding was that Word2Vec[1] was trained on Wikipedia and other such texts, not artificially constructed things like the triplets you suggest. There's an inherent structure present in human languages that enable the "magic" of embeddings to work, as far as I can tell. [1] https://code.google.com/archive/p/word2vec/ https://code.google.com/archive/p/word2vec/

itronitron 3y ago
Has there been any rigorous evaluation of word2vec calculating 'king-man+woman=queen' associations? I only recall the author providing some cherry-picked examples from their results, which I suppose makes it a seminal AI paper.
- kevindamm 3y ago
  The original paper included source, and that has their test data and results -- it gets ~77% accuracy on about 20k example word analogies (with 99.7% coverage), and 78% accuracy with phrases with 77% coverage (and a much smaller data set, 3,218 examples). You can see the test sets here: https://github.com/tmikolov/word2vec/blob/master/questions-words.txt https://github.com/tmikolov/word2vec/blob/master/questions-w... https://github.com/tmikolov/word2vec/blob/master/questions-phrases.txt https://github.com/tmikolov/word2vec/blob/master/questions-p... and you could see how much better LLMs do on the same 20k examples.
- sp332 3y ago
  A lot more info about how it worked here https://blog.esciencecenter.nl/king-man-woman-king-9a7fd2935a85 https://blog.esciencecenter.nl/king-man-woman-king-9a7fd2935... and an interactive demo https://dash.gallery/dash-word-arithmetic/ https://dash.gallery/dash-word-arithmetic/ (blog post about the demo https://medium.com/plotly/understanding-word-embedding-arithmetic-why-theres-no-single-answer-to-king-man-woman-cd2760e2cb7f https://medium.com/plotly/understanding-word-embedding-arith...)
- 0xdeadbeefbabe 3y ago
  If a paper attacks plus and minus it ought to cite this one.
kevindamm 3y ago
Ah, yes I may have mis-remembered or misunderstood when looking at the training data and model definition, it would have been about a decade ago now. Or perhaps I was thinking of an unrelated experiment to use chosen analogies during training but yeah that isn't part of the original paper. The training process from the original Mikolov et al. paper only uses the analogy examples (questions-words.txt and questions-phrases.txt) to measure accuracy after training: https://github.com/tmikolov/word2vec/blob/20c129af10659f7c50e86e3be406df663beff438/demo-train-big-model-v1.sh#L98 https://github.com/tmikolov/word2vec/blob/20c129af10659f7c50...
_t89y 3y ago
There is an inherent structure in language. Embeddings do not and will not capture it. It's why they do not work. Their ability to form grammatical sentences with high accuracy is part of the illusion that you have been understood.