2 ms·

Has there been any rigorous evaluation of word2vec calculating 'king-man+woman=queen' associations? I only recall the author providing some cherry-picked exampl

• kevindamm 3y ago
  The original paper included source, and that has their test data and results -- it gets ~77% accuracy on about 20k example word analogies (with 99.7% coverage), and 78% accuracy with phrases with 77% coverage (and a much smaller data set, 3,218 examples). You can see the test sets here: https://github.com/tmikolov/word2vec/blob/master/questions-words.txt https://github.com/tmikolov/word2vec/blob/master/questions-w... https://github.com/tmikolov/word2vec/blob/master/questions-phrases.txt https://github.com/tmikolov/word2vec/blob/master/questions-p... and you could see how much better LLMs do on the same 20k examples.
• sp332 3y ago
  A lot more info about how it worked here https://blog.esciencecenter.nl/king-man-woman-king-9a7fd2935a85 https://blog.esciencecenter.nl/king-man-woman-king-9a7fd2935... and an interactive demo https://dash.gallery/dash-word-arithmetic/ https://dash.gallery/dash-word-arithmetic/ (blog post about the demo https://medium.com/plotly/understanding-word-embedding-arithmetic-why-theres-no-single-answer-to-king-man-woman-cd2760e2cb7f https://medium.com/plotly/understanding-word-embedding-arith...)
• 0xdeadbeefbabe 3y ago
  If a paper attacks plus and minus it ought to cite this one.