6 ms·
Sense2vec – A Fast and Accurate Method for Word Sense Disambiguation
- n0us 11y agoAnyone know of an open source implementation? I've only had a chance to scan the document but it appears to only go into the theory.
- striking 11y agoIt appears they're only releasing the theory, that the implementation is up to us. Which, given the time they've put into this paper, is not an unreasonable thing. Edit: just noticed their email addresses are of the http://www.digitalreasoning.com/ http://www.digitalreasoning.com/ domain. Seems they're not in the business of releasing open source versions, so to speak. OS versions of similar tech exist, but this is new.
- syllogism 11y agoWe'll have this implemented in spaCy before too long. It's actually super easy to do --- all you need is to merge the part of speech tags or entity labels onto the tokens before feeding the text to Gensim or another word2vec implementation. I've wanted to do this for a while, so it's nice to see that it works well.
- danieldk 11y agoIndeed, the implementation is very simple. We had considered this idea some times as well, but like you didn't get around to implement/evaluate it yet. So, it's good to hear that it works. One thing notably absent from the paper is a discussion of the trade-off between augmenting tokens with annotations in this way for sense disambiguation vs. data sparseness. Their approach may make the embeddings for frequent senses better, but the difficulty in WSD is typically in low-frequency senses. I think that particularly in disambiguation using part-of-speech tags, there is still a high semantic relatedness between senses, especially in languages with frequent nominalization or verbalization. I can imagine that an model that predicts a target (or context) in decomposed form (token and label) might improve embeddings for low-frequent senses.
- syllogism 11y agoThis isn't really WSD though, or at least, only very weakly. Rare words are usually pretty unambiguous for part-of-speech. I would guess this mostly has an effect on the top 5,000 items of the vocabulary, and most of the rest of the lexicon only has a single "sense".
- danieldk 11y agoThis isn't really WSD though, or at least, only very weakly. Sure. I was pointing to real WSD where sparseness becomes even a stronger problem than when your definition of sense is restricted to part-of-speech tag or sentiment. Rare words are usually pretty unambiguous for part-of-speech. I was talking about (possibly) frequent words where some part-of-speech are infrequent, not about rare words. To take five more or less random examples form the Brown corpus (yes, we train on large corpora, but I think similar distributions could hold for less frequent forms in languages with e.g. frequent nomalization, not everyone speaks English!): mother NN 173 VB 1 code NN 20 VB 1 hanging NN 1 VBG 20 level JJ 14 NN 172 VB 2 services NNS 115 VBZ 1 If your learning method is as coarse-grained as simply throwing the token plus part-of-speech into word2vec or wang2vec, some will be below the frequency cut-offs (or will be to sparse to learn good embeddings), while other 'senses' may in reality be semantically similar.
- syllogism 11y agoThanks for the explanation. I see what you're saying now.
- benten10 11y agoThis is really excellent for researchers approaching the field from the more NLP angle than the ML/CompLing angle. It was only a few weeks(days?) ago that Chris Manning came up with the whitepaper arguing that linguists have just as much/even more relevance now and shouldn't worry that neural networks will eat their lunch, and then we get to see this. This is awesome! : )
- YeGoblynQueenne 11y agoHave you a link to that whitepaper? I found an uncorrected draft, but I'd like to read the finished thing- cheers.
- williamtrask 11y agoi think this one: http://www.mitpressjournals.org/doi/pdf/10.1162/COLI_a_00239 http://www.mitpressjournals.org/doi/pdf/10.1162/COLI_a_00239
- YeGoblynQueenne 11y agoThat's the draft one but thank you anyway :)
- williamtrask 11y agoworking on it. we'll see.
- jilebedev 11y agoFor a layman's introduction to how (pardon the hyperbole) soul-crushingly difficult this problem is, have a look at this amateur attempt to process language inputted by players into a video game: https://www.youtube.com/watch?v=Ff6V1yFafW4 https://www.youtube.com/watch?v=Ff6V1yFafW4
- habitue 11y agoSupervised disambiguation? Isn't the entire reason word2vec is exciting because it's unsupervised?
- accurrent 11y agoNot the only reason though. Its exciting because it turns a word into a reasonably sized vector. After that the vector can be fed into neural networks (nueral networks take vectors for input) or operations can be performed. Like Paris - France ~= Beijing - China.
- ya3r 11y agoI have seen this many times that people claim the word2vec is unsupervised. But I think that is inaccurate. word2vec is using a very weak supervision which is the order in which words appear in a meaningful sentence. And I think it is fascinating to use this kind of weak supervision to build distributed embedding for words.
- Matumio 11y agoThis makes me wonder if the supervised/unsupervised distinction is useful at all. Maybe you could say instead that all learning algorithms simply need some way to measure similarity between the training examples, and it doesn't make a fundamental difference if you cluster examples by their target label ("supervised") or by their input vectors ("unsupervised") or by the context in which they appear (word2vec).
- fnl 11y agoSkip-grams only take focus word, context word pairs. So those pairs do not take order into account
- ya3r 11y agobeing in the same context is some kind of order.
- 11y ago
- j_jochem 11y ago> We demonstrate that these embeddings can disambiguate both contrastive senses such as nominal and verbal senses as well as nuanced senses such as sarcasm. If this is true, that's astonishing. Also, this might allow us to build assistive technologies for people who are unable to perceive sarcasm.
- syllogism 11y agoLike a lot of work in NLP at the moment, this is a reasonably straight-forward mash-up of existing techniques. This particular idea is pretty obvious. What wasn't obvious was whether it would work well, which was why nobody else got around to trying it yet. The experiments are nicely conducted, with strong baselines across multiple evaluations. The authors also include experiments with variations of the idea, to further validate the approach. To understand the technique, first understand word2vec: http://rare-technologies.com/word2vec-tutorial/ http://rare-technologies.com/word2vec-tutorial/ http://colah.github.io/posts/2014-07-NLP-RNNs-Representations/ http://colah.github.io/posts/2014-07-NLP-RNNs-Representation... Now understand part-of-speech tagging: http://spacy.io/blog/part-of-speech-POS-tagger-in-python/ http://spacy.io/blog/part-of-speech-POS-tagger-in-python/ By default word2vec gives you clusters for each word, this paper is giving you clusters for word_POS, e.g. The_DT apple_NNP employee_NN is_VBZ eating_VBG an_DT apple_NN. The same trick is done with named entity labels as well. The following papers explain how the new word vectors are used in a dependency parser: Collobert and Weston (2011): http://arxiv.org/pdf/1103.0398.pdf http://arxiv.org/pdf/1103.0398.pdf Wang and Manning (2014): http://cs.stanford.edu/~danqi/papers/emnlp2014.pdf http://cs.stanford.edu/~danqi/papers/emnlp2014.pdf Yoav Goldberg (2015): http://u.cs.biu.ac.il/~yogo/nnlp.pdf http://u.cs.biu.ac.il/~yogo/nnlp.pdf Survey/review, aimed at grad students
- abtinf 11y ago"This particular idea is pretty obvious. What wasn't obvious was whether it would work well, which was why nobody else got around to trying it yet." Ideas are often obvious after you hear them for the first time.
- bhickey 11y agoThis one is actually kind-of, sort-of obvious. For NaNoGenMo this year I smushed together word2vec and a POS tagger. What the authors have done here is really cool and goes miles beyond my hacks, but the kernel of the idea should be obvious to anyone familiar with word2vec.
- deepGem 11y agoIf you really want to understand word2vec watch CS224D second lecture. https://www.youtube.com/watch?v=T8tQZChniMk https://www.youtube.com/watch?v=T8tQZChniMk
- andrewtbham 11y agoI think this paper is interesting... The section on sarcasm is interesting. They got good results distinguishing apple the noun (which is similar to apples, pear, peach, blueberry) from Apple the proper noun (which is similar to Microsoft, iphone, ipad, samsung.) While they got good results telling bank the noun from bank the verb... they didn't differentiate bank the noun (financial ) from bank the noun (the side of a river). Or even more complicated... look at all the uses of bank on word net. http://wordnetweb.princeton.edu/perl/webwn?s=bank http://wordnetweb.princeton.edu/perl/webwn?s=bank I'm not sure I can grasp the implications of disambiguating some words, but not others. For some applications it might make sense but... trying to disambiguate words or leave them as single vectors is still, in my mind, an open research question.
- williamtrask 11y agoGreat point. In my opinion, for improving the quality of syntactic tasks such as POS / dependency parse, the difference between disambiguating and not disambiguating riverbank and financial bank will be minimal. However, for semantic tasks (perhaps NER, information extraction, question answering) the difference would be more profound. This paper is primarily focused on a much more efficient method to do the former.
- fnl 11y agoWhat always worries me with all WSD approaches is the performance tradeoff: How much more performance is gained from using more complex per-sense word vectors designs vs. "standard" word embeddings. Setup complexity can often increase significantly for these models and training times are much longer, while the gains from these approaches are not terribly clear to date.
- williamtrask 11y agoIn this case, there is no performance tradeoff except that of running your core NLP pipeline... for which there are several very fast options
- fnl 11y agoSo your results demonstrate that directed dependency labeling works better with vectors learned from PoS-tagged words than with PoS-tagged vectors (learned from untagged words)? And if so, why are you sure you are not overfitting on the corpus or that the "unseen" (in your case: label +) word (pairs) issues will in the end do more harm than what you gain when using this approach on truly independent data/text? EDIT: Sorry, this question above is probably too convoluted to understand. As I understand, the evaluation of the UAS in the paper was made by letting the parser use the gold PoS labels from the UD treebank (plus either word embeddings). But what would happen if the PoS labels for evaluating the dependency parser came from a PoS tagger, as would be the case when working on unseen data? I might imagine that "plain" embeddings could maybe produce a better UAS in that case, because they are not as "overfitted" as the "enriched" embeddings (as those are derived from the PoS-tagger labeled words in the first place).
- williamtrask 11y agoPerhaps, but seeing as POS taggers are ~97% accurate (at least in English), I'd expect this to be minimal. Furthermore, the baseline neural network also has access to the gold standard POS tags, so the comparison of adding POS disambiguated embeddings is pretty clean. It's the difference between "words + pos tags" as features and "pos-disambiguated word + pos tags".