3 ms·
The problem is that Word2Vec and Glove never really worked from an engineering perspective but there was widespread belief that they did. From a scientific per
by PaulHoule 2y ago
The problem is that Word2Vec and Glove never really worked from an engineering perspective but there was widespread belief that they did.
From a scientific perspective there are biases in the way that people do science that let an "emperor has no clothes" situation go on too long.
For instance, I never saw the negative result that "you can't build a classifier for words" based on word vectors published, because people don't publish negative results. They assume they were doing something wrong, there were just too many blog posts (not papers) were people were repeating the lucky result that king - women + man = queen [1]. Anyone who looked at it systematically found it was a morass, got bogged down, and didn't publish anything. It's pretty common for fields to get stuck in a place like this and make no progress or phony progress [2]. Nobody published "a classifier for words" for Word2Vec because it doesn't work, even though it tantalizingly seems to work when N is very small because of the way high dimensional spaces work.
Blog posts like this one will perpetuate more people getting stuck in a morass and leave them confused, writing more blog posts that repeat the little bit that seems to make sense. Thus being able to use language and be part of a group is a curse and not a blessing because it makes you repeat other people's mistakes not learn from them.
[1] yes, they did learn to do things like this better than chance, but chance is pretty bad
[2] https://www.stat.cmu.edu/~cshalizi/2010-10-18-Meetup.pdf https://www.stat.cmu.edu/~cshalizi/2010-10-18-Meetup.pdf