3 ms·
I think I have an intuitive explanation why word2vec works. I tried to write it down, but I'm lazy. Don't think of it as a 2 layer neural network. what it real
by billconan 11y ago
I think I have an intuitive explanation why word2vec works. I tried to write it down, but I'm lazy.
Don't think of it as a 2 layer neural network. what it really is is a physics system. like a universe where each word is a planet. when words appear in the same context, that will assign some gravitational pull to the "planets". eventually, words with with same meaning will clustered as a galaxy.
for example, look at the following 2 sentences:
1. I pat a dog.
2. I pat a cat.
dog and cat, they appear in the same context. when the first time we see the word dog, we calculate center of mass of "I", "pat" and "a". Think of them as little planets.
then you give the planet "dog" a little force to pull it to the center of mass of the previous three words.
when you see the word cat, you do the same as it's in the same context.
Eventually, cat and dog will be clustered and form a galaxy, with other similar words. and the galaxy will be called "pet".
- LordRegex 11y agohttp://lesswrong.com/lw/nl/the_cluster_structure_of_thingspace/ http://lesswrong.com/lw/nl/the_cluster_structure_of_thingspa...
- robrenaud 11y agoI don't know that they physics analogy works so well, or at least, it's definitely missing something. What prevents the whole word "universe" from collapsing on itself, forming a black hole? That is, if there are only attractive forces, the global optimum is to co-locate everything in the same point, which doesn't give you a useful model. There needs to be something in the model that keeps different words apart from each other. This page has the clearest explanation of word embeddings and the relationship between the objective function and why vector translation captures meaning. http://nlp.stanford.edu/projects/glove/ http://nlp.stanford.edu/projects/glove/
- billconan 11y agoit works because the gravity of word2vec isn't the gravity of real life. notice that I only pull the word "dog" to the center of gravity of the rest words, instead of pulling all of them together. I think the full version even push the rest of the words away from the center of gravity. but I need to double check the math. this is not just an analogy. this is what word2vec's math says. the only analogy part is that word2vec is in high dimension, my analogy is in 3 dimension.
- deleted 11y ago[deleted]
- billconan 11y agoin real world, gravitational force appears on every object. in the word2vec world, it only appears on words found in similar context. that's one major difference. I think there is also anti-gravitational force that pushes words away. but again I need to double check.