4 ms·

I think the article overestimates how much we understand about neural networks, and the novel ways we might use them. Take word2vec for example. It's a two-lay

I think the article overestimates how much we understand about neural networks, and the novel ways we might use them. Take word2vec for example. It's a two-layer (i.e. not deep) neural network that uses a relatively simple training algorithm. Yet, after 30 minutes it can learn impressive relationships between English words. The method is only 3 years old and businesses continue to find novel ways to apply word2vec. Check out this article: http://multithreaded.stitchfix.com/blog/2015/03/11/word-is-worth-a-thousand-vectors/ http://multithreaded.stitchfix.com/blog/2015/03/11/word-is-w... They are using word2vec to find clothing items that resemble other products by adding or subtracting descriptors such as "stripes", "blue", or even "pregnant." For my company, we are attempting to normalize medical terminology using word2vec. Here are a couple examples: 1. "myocardial" + "infarction" = "MI" 2. "ring" + "finger" = "fourth" + "metacarpal" We never would have thought to normalize the names of different body parts, but with word2vec, it does it for us. There's a great deal yet to learn about how this model alone works, let alone deeper networks.

billconan 11y ago
I think I have an intuitive explanation why word2vec works. I tried to write it down, but I'm lazy. Don't think of it as a 2 layer neural network. what it really is is a physics system. like a universe where each word is a planet. when words appear in the same context, that will assign some gravitational pull to the "planets". eventually, words with with same meaning will clustered as a galaxy. for example, look at the following 2 sentences: 1. I pat a dog. 2. I pat a cat. dog and cat, they appear in the same context. when the first time we see the word dog, we calculate center of mass of "I", "pat" and "a". Think of them as little planets. then you give the planet "dog" a little force to pull it to the center of mass of the previous three words. when you see the word cat, you do the same as it's in the same context. Eventually, cat and dog will be clustered and form a galaxy, with other similar words. and the galaxy will be called "pet".
- LordRegex 11y ago
  http://lesswrong.com/lw/nl/the_cluster_structure_of_thingspace/ http://lesswrong.com/lw/nl/the_cluster_structure_of_thingspa...
- robrenaud 11y ago
  I don't know that they physics analogy works so well, or at least, it's definitely missing something. What prevents the whole word "universe" from collapsing on itself, forming a black hole? That is, if there are only attractive forces, the global optimum is to co-locate everything in the same point, which doesn't give you a useful model. There needs to be something in the model that keeps different words apart from each other. This page has the clearest explanation of word embeddings and the relationship between the objective function and why vector translation captures meaning. http://nlp.stanford.edu/projects/glove/ http://nlp.stanford.edu/projects/glove/
  - billconan 11y ago
    it works because the gravity of word2vec isn't the gravity of real life. notice that I only pull the word "dog" to the center of gravity of the rest words, instead of pulling all of them together. I think the full version even push the rest of the words away from the center of gravity. but I need to double check the math. this is not just an analogy. this is what word2vec's math says. the only analogy part is that word2vec is in high dimension, my analogy is in 3 dimension.
    - deleted 11y ago
      [deleted]
  - billconan 11y ago
    in real world, gravitational force appears on every object. in the word2vec world, it only appears on words found in similar context. that's one major difference. I think there is also anti-gravitational force that pushes words away. but again I need to double check.
varelse 11y ago
This approach works IMO only when all of the terms have sufficient representation in the dataset. I have noticed that Google seems to have recently incorporated something like this as one of its scoring factors. That's fine if you're searching for the latest Kardashian faux pas (and such searches dominate the use of Google IMO hence the results of any A/B test I suspect), but my searches for obscure technical terms have been returning much less relevant results than they once did. Has anyone else noticed this?
- douche 11y ago
  I suspect the indexers/spiders don't work as well as they once did. Routinely I've been getting results for sketchy mirrors of StackOverflow or the MSDN forums, which link back to the original MSDN or SO post, but that original is nowhere to be found in the Google results.
sp3ktr 11y ago
word2vec has been shown to be completely equal to a much older method so it's not as mysterious or new as one at first usually believes. Source (great paper): http://u.cs.biu.ac.il/~nlp/wp-content/uploads/Neural-Word-Embeddings-as-Implicit-Matrix-Factorization-NIPS-2014.pdf http://u.cs.biu.ac.il/~nlp/wp-content/uploads/Neural-Word-Em...
tamana 11y ago
Those examples aren't vectors, they are synonyms. Does your system find "ring toes"?

billconan 11y ago

I think I have an intuitive explanation why word2vec works. I tried to write it down, but I'm lazy. Don't think of it as a 2 layer neural network. what it really is is a physics system. like a universe where each word is a planet. when words appear in the same context, that will assign some gravitational pull to the "planets". eventually, words with with same meaning will clustered as a galaxy. for example, look at the following 2 sentences: 1. I pat a dog. 2. I pat a cat. dog and cat, they appear in the same context. when the first time we see the word dog, we calculate center of mass of "I", "pat" and "a". Think of them as little planets. then you give the planet "dog" a little force to pull it to the center of mass of the previous three words. when you see the word cat, you do the same as it's in the same context. Eventually, cat and dog will be clustered and form a galaxy, with other similar words. and the galaxy will be called "pet".

LordRegex 11y ago
http://lesswrong.com/lw/nl/the_cluster_structure_of_thingspace/ http://lesswrong.com/lw/nl/the_cluster_structure_of_thingspa...
robrenaud 11y ago
I don't know that they physics analogy works so well, or at least, it's definitely missing something. What prevents the whole word "universe" from collapsing on itself, forming a black hole? That is, if there are only attractive forces, the global optimum is to co-locate everything in the same point, which doesn't give you a useful model. There needs to be something in the model that keeps different words apart from each other. This page has the clearest explanation of word embeddings and the relationship between the objective function and why vector translation captures meaning. http://nlp.stanford.edu/projects/glove/ http://nlp.stanford.edu/projects/glove/
- billconan 11y ago
  it works because the gravity of word2vec isn't the gravity of real life. notice that I only pull the word "dog" to the center of gravity of the rest words, instead of pulling all of them together. I think the full version even push the rest of the words away from the center of gravity. but I need to double check the math. this is not just an analogy. this is what word2vec's math says. the only analogy part is that word2vec is in high dimension, my analogy is in 3 dimension.
  - deleted 11y ago
    [deleted]
- billconan 11y ago
  in real world, gravitational force appears on every object. in the word2vec world, it only appears on words found in similar context. that's one major difference. I think there is also anti-gravitational force that pushes words away. but again I need to double check.

varelse 11y ago

This approach works IMO only when all of the terms have sufficient representation in the dataset. I have noticed that Google seems to have recently incorporated something like this as one of its scoring factors. That's fine if you're searching for the latest Kardashian faux pas (and such searches dominate the use of Google IMO hence the results of any A/B test I suspect), but my searches for obscure technical terms have been returning much less relevant results than they once did. Has anyone else noticed this?

douche 11y ago
I suspect the indexers/spiders don't work as well as they once did. Routinely I've been getting results for sketchy mirrors of StackOverflow or the MSDN forums, which link back to the original MSDN or SO post, but that original is nowhere to be found in the Google results.

sp3ktr 11y ago

word2vec has been shown to be completely equal to a much older method so it's not as mysterious or new as one at first usually believes. Source (great paper): http://u.cs.biu.ac.il/~nlp/wp-content/uploads/Neural-Word-Embeddings-as-Implicit-Matrix-Factorization-NIPS-2014.pdf http://u.cs.biu.ac.il/~nlp/wp-content/uploads/Neural-Word-Em...

tamana 11y ago

Those examples aren't vectors, they are synonyms. Does your system find "ring toes"?