3 ms·

Not to be overly pedantic, but this comment misses one major important delineation between engineering and science. The old adage is that “with engineering, if

Not to be overly pedantic, but this comment misses one major important delineation between engineering and science. The old adage is that “with engineering, if you don’t know what you’re doing; you’re doing it wrong. With science, if you know what you’re doing; you’re doing it wrong.” The older models should not be forgotten, since science - when done appropriately - relies on incremental improvements and experimentation. However, for the pure purposes of building something, if the modern ways are an improvement on the old in most or all measurable ways - sure there probably isn’t much prudence beyond novelty to utilize the outdated methods.

PaulHoule 2y ago
The problem is that Word2Vec and Glove never really worked from an engineering perspective but there was widespread belief that they did. From a scientific perspective there are biases in the way that people do science that let an "emperor has no clothes" situation go on too long. For instance, I never saw the negative result that "you can't build a classifier for words" based on word vectors published, because people don't publish negative results. They assume they were doing something wrong, there were just too many blog posts (not papers) were people were repeating the lucky result that king - women + man = queen [1]. Anyone who looked at it systematically found it was a morass, got bogged down, and didn't publish anything. It's pretty common for fields to get stuck in a place like this and make no progress or phony progress [2]. Nobody published "a classifier for words" for Word2Vec because it doesn't work, even though it tantalizingly seems to work when N is very small because of the way high dimensional spaces work. Blog posts like this one will perpetuate more people getting stuck in a morass and leave them confused, writing more blog posts that repeat the little bit that seems to make sense. Thus being able to use language and be part of a group is a curse and not a blessing because it makes you repeat other people's mistakes not learn from them. [1] yes, they did learn to do things like this better than chance, but chance is pretty bad [2] https://www.stat.cmu.edu/~cshalizi/2010-10-18-Meetup.pdf https://www.stat.cmu.edu/~cshalizi/2010-10-18-Meetup.pdf

PaulHoule 2y ago

The problem is that Word2Vec and Glove never really worked from an engineering perspective but there was widespread belief that they did. From a scientific perspective there are biases in the way that people do science that let an "emperor has no clothes" situation go on too long. For instance, I never saw the negative result that "you can't build a classifier for words" based on word vectors published, because people don't publish negative results. They assume they were doing something wrong, there were just too many blog posts (not papers) were people were repeating the lucky result that king - women + man = queen [1]. Anyone who looked at it systematically found it was a morass, got bogged down, and didn't publish anything. It's pretty common for fields to get stuck in a place like this and make no progress or phony progress [2]. Nobody published "a classifier for words" for Word2Vec because it doesn't work, even though it tantalizingly seems to work when N is very small because of the way high dimensional spaces work. Blog posts like this one will perpetuate more people getting stuck in a morass and leave them confused, writing more blog posts that repeat the little bit that seems to make sense. Thus being able to use language and be part of a group is a curse and not a blessing because it makes you repeat other people's mistakes not learn from them. [1] yes, they did learn to do things like this better than chance, but chance is pretty bad [2] https://www.stat.cmu.edu/~cshalizi/2010-10-18-Meetup.pdf https://www.stat.cmu.edu/~cshalizi/2010-10-18-Meetup.pdf