4 ms·

Like a lot of work in NLP at the moment, this is a reasonably straight-forward mash-up of existing techniques. This particular idea is pretty obvious. What wasn

Like a lot of work in NLP at the moment, this is a reasonably straight-forward mash-up of existing techniques. This particular idea is pretty obvious. What wasn't obvious was whether it would work well, which was why nobody else got around to trying it yet. The experiments are nicely conducted, with strong baselines across multiple evaluations. The authors also include experiments with variations of the idea, to further validate the approach. To understand the technique, first understand word2vec: http://rare-technologies.com/word2vec-tutorial/ http://rare-technologies.com/word2vec-tutorial/ http://colah.github.io/posts/2014-07-NLP-RNNs-Representations/ http://colah.github.io/posts/2014-07-NLP-RNNs-Representation... Now understand part-of-speech tagging: http://spacy.io/blog/part-of-speech-POS-tagger-in-python/ http://spacy.io/blog/part-of-speech-POS-tagger-in-python/ By default word2vec gives you clusters for each word, this paper is giving you clusters for word_POS, e.g. The_DT apple_NNP employee_NN is_VBZ eating_VBG an_DT apple_NN. The same trick is done with named entity labels as well. The following papers explain how the new word vectors are used in a dependency parser: Collobert and Weston (2011): http://arxiv.org/pdf/1103.0398.pdf http://arxiv.org/pdf/1103.0398.pdf Wang and Manning (2014): http://cs.stanford.edu/~danqi/papers/emnlp2014.pdf http://cs.stanford.edu/~danqi/papers/emnlp2014.pdf Yoav Goldberg (2015): http://u.cs.biu.ac.il/~yogo/nnlp.pdf http://u.cs.biu.ac.il/~yogo/nnlp.pdf Survey/review, aimed at grad students

abtinf 11y ago
"This particular idea is pretty obvious. What wasn't obvious was whether it would work well, which was why nobody else got around to trying it yet." Ideas are often obvious after you hear them for the first time.
- bhickey 11y ago
  This one is actually kind-of, sort-of obvious. For NaNoGenMo this year I smushed together word2vec and a POS tagger. What the authors have done here is really cool and goes miles beyond my hacks, but the kernel of the idea should be obvious to anyone familiar with word2vec.
deepGem 11y ago
If you really want to understand word2vec watch CS224D second lecture. https://www.youtube.com/watch?v=T8tQZChniMk https://www.youtube.com/watch?v=T8tQZChniMk
benten10 11y ago
Since this appears to be the 'related works' thread, I'll add my links here. A reasonable amount of work has already been done in disambiguation of the meanings of words with word embeddings, some of it trained, some untrained. These particular papers that have recently caught my attention, and have the potential to bring significant gains wrt existing technologies are these: 1. Infinite-Length Word embeddings http://arxiv.org/pdf/1511.05392v2.pdf http://arxiv.org/pdf/1511.05392v2.pdf I am particularly excited about this paper, as the authors suggest a method to create variable-length word vectors, depending on the multiplicity of meanings a word can have (so theoretically, if a word had a _lot_ of meanings, the word vectors would me very large. Reminds one of the word 'Aladeen'). IF the authors happen to me hanging out here, the community is eagerly waiting for the reference implementation. : ) 2. Joint Learning of Words and Meaning Representations for Open-Text Semantic Parsing http://jmlr.csail.mit.edu/proceedings/papers/v22/bordes12/bordes12.pdf http://jmlr.csail.mit.edu/proceedings/papers/v22/bordes12/bo... 3. Breaking Sticks And Ambiguities With Adaptive Skip-Gram http://arxiv.org/pdf/1502.07257v2.pdf http://arxiv.org/pdf/1502.07257v2.pdf Edit: added paper 3.
Udik 11y ago
How does this help with words, for example substantives, that can have completely different meanings depending on the context? Ex: The ball is bouncing on the floor The ball takes place in the hotel
- nl 11y ago
  Not sure this will help much here, because in both of those cases "ball" is a noun. However, plain "old" word2vec will handle it fine. A round ball is associated with words like bounce, and various sporting terms. It will appear in a the vector space alongside those words. A dance ball is associated with dance, gowns, "go to", etc. The example in the paper around "Washington" shows how this can be used to distinguish between the NER (named entity resolution) type "PERSON" and "GPE" (Geo-political-entity) - it disambiguates very well between Washington (George) and Washington (DC).
  - Udik 11y ago
    Hmm, but isn't word2vec a dictionary in word -> vector? If it's the case, the two - completely unrelated - meanings of the word "ball" will be represented by a single vector. Which would mean that in that representation, for example, the concept of a ball is near both to "sphere" and "music" or "dancing". So that the perfect ball is a dancing sphere, or something like that.

abtinf 11y ago

"This particular idea is pretty obvious. What wasn't obvious was whether it would work well, which was why nobody else got around to trying it yet." Ideas are often obvious after you hear them for the first time.

bhickey 11y ago
This one is actually kind-of, sort-of obvious. For NaNoGenMo this year I smushed together word2vec and a POS tagger. What the authors have done here is really cool and goes miles beyond my hacks, but the kernel of the idea should be obvious to anyone familiar with word2vec.

deepGem 11y ago

If you really want to understand word2vec watch CS224D second lecture. https://www.youtube.com/watch?v=T8tQZChniMk https://www.youtube.com/watch?v=T8tQZChniMk

benten10 11y ago

Since this appears to be the 'related works' thread, I'll add my links here. A reasonable amount of work has already been done in disambiguation of the meanings of words with word embeddings, some of it trained, some untrained. These particular papers that have recently caught my attention, and have the potential to bring significant gains wrt existing technologies are these: 1. Infinite-Length Word embeddings http://arxiv.org/pdf/1511.05392v2.pdf http://arxiv.org/pdf/1511.05392v2.pdf I am particularly excited about this paper, as the authors suggest a method to create variable-length word vectors, depending on the multiplicity of meanings a word can have (so theoretically, if a word had a _lot_ of meanings, the word vectors would me very large. Reminds one of the word 'Aladeen'). IF the authors happen to me hanging out here, the community is eagerly waiting for the reference implementation. : ) 2. Joint Learning of Words and Meaning Representations for Open-Text Semantic Parsing http://jmlr.csail.mit.edu/proceedings/papers/v22/bordes12/bordes12.pdf http://jmlr.csail.mit.edu/proceedings/papers/v22/bordes12/bo... 3. Breaking Sticks And Ambiguities With Adaptive Skip-Gram http://arxiv.org/pdf/1502.07257v2.pdf http://arxiv.org/pdf/1502.07257v2.pdf Edit: added paper 3.

Udik 11y ago

How does this help with words, for example substantives, that can have completely different meanings depending on the context? Ex: The ball is bouncing on the floor The ball takes place in the hotel

nl 11y ago
Not sure this will help much here, because in both of those cases "ball" is a noun. However, plain "old" word2vec will handle it fine. A round ball is associated with words like bounce, and various sporting terms. It will appear in a the vector space alongside those words. A dance ball is associated with dance, gowns, "go to", etc. The example in the paper around "Washington" shows how this can be used to distinguish between the NER (named entity resolution) type "PERSON" and "GPE" (Geo-political-entity) - it disambiguates very well between Washington (George) and Washington (DC).
- Udik 11y ago
  Hmm, but isn't word2vec a dictionary in word -> vector? If it's the case, the two - completely unrelated - meanings of the word "ball" will be represented by a single vector. Which would mean that in that representation, for example, the concept of a ball is near both to "sphere" and "music" or "dancing". So that the perfect ball is a dancing sphere, or something like that.