4 ms·

Anyone know of an open source implementation? I've only had a chance to scan the document but it appears to only go into the theory.

by n0us 11y ago

striking 11y ago
It appears they're only releasing the theory, that the implementation is up to us. Which, given the time they've put into this paper, is not an unreasonable thing. Edit: just noticed their email addresses are of the http://www.digitalreasoning.com/ http://www.digitalreasoning.com/ domain. Seems they're not in the business of releasing open source versions, so to speak. OS versions of similar tech exist, but this is new.
syllogism 11y ago
We'll have this implemented in spaCy before too long. It's actually super easy to do --- all you need is to merge the part of speech tags or entity labels onto the tokens before feeding the text to Gensim or another word2vec implementation. I've wanted to do this for a while, so it's nice to see that it works well.
- danieldk 11y ago
  Indeed, the implementation is very simple. We had considered this idea some times as well, but like you didn't get around to implement/evaluate it yet. So, it's good to hear that it works. One thing notably absent from the paper is a discussion of the trade-off between augmenting tokens with annotations in this way for sense disambiguation vs. data sparseness. Their approach may make the embeddings for frequent senses better, but the difficulty in WSD is typically in low-frequency senses. I think that particularly in disambiguation using part-of-speech tags, there is still a high semantic relatedness between senses, especially in languages with frequent nominalization or verbalization. I can imagine that an model that predicts a target (or context) in decomposed form (token and label) might improve embeddings for low-frequent senses.
  - syllogism 11y ago
    This isn't really WSD though, or at least, only very weakly. Rare words are usually pretty unambiguous for part-of-speech. I would guess this mostly has an effect on the top 5,000 items of the vocabulary, and most of the rest of the lexicon only has a single "sense".
    - danieldk 11y ago
      This isn't really WSD though, or at least, only very weakly. Sure. I was pointing to real WSD where sparseness becomes even a stronger problem than when your definition of sense is restricted to part-of-speech tag or sentiment. Rare words are usually pretty unambiguous for part-of-speech. I was talking about (possibly) frequent words where some part-of-speech are infrequent, not about rare words. To take five more or less random examples form the Brown corpus (yes, we train on large corpora, but I think similar distributions could hold for less frequent forms in languages with e.g. frequent nomalization, not everyone speaks English!): mother NN 173 VB 1 code NN 20 VB 1 hanging NN 1 VBG 20 level JJ 14 NN 172 VB 2 services NNS 115 VBZ 1 If your learning method is as coarse-grained as simply throwing the token plus part-of-speech into word2vec or wang2vec, some will be below the frequency cut-offs (or will be to sparse to learn good embeddings), while other 'senses' may in reality be semantically similar.
      - syllogism 11y ago
        Thanks for the explanation. I see what you're saying now.
- benten10 11y ago
  This is really excellent for researchers approaching the field from the more NLP angle than the ML/CompLing angle. It was only a few weeks(days?) ago that Chris Manning came up with the whitepaper arguing that linguists have just as much/even more relevance now and shouldn't worry that neural networks will eat their lunch, and then we get to see this. This is awesome! : )
  - YeGoblynQueenne 11y ago
    Have you a link to that whitepaper? I found an uncorrected draft, but I'd like to read the finished thing- cheers.
    - williamtrask 11y ago
      i think this one: http://www.mitpressjournals.org/doi/pdf/10.1162/COLI_a_00239 http://www.mitpressjournals.org/doi/pdf/10.1162/COLI_a_00239
      - YeGoblynQueenne 11y ago
        That's the draft one but thank you anyway :)
williamtrask 11y ago
working on it. we'll see.

Anyone know of an open source implementation? I've only had a chance to scan the document but it appears to only go into the theory.

by n0us 11y ago

Anyone know of an open source implementation? I've only had a chance to scan the document but it appears to only go into the theory.

striking 11y ago

It appears they're only releasing the theory, that the implementation is up to us. Which, given the time they've put into this paper, is not an unreasonable thing. Edit: just noticed their email addresses are of the http://www.digitalreasoning.com/ http://www.digitalreasoning.com/ domain. Seems they're not in the business of releasing open source versions, so to speak. OS versions of similar tech exist, but this is new.

syllogism 11y ago

We'll have this implemented in spaCy before too long. It's actually super easy to do --- all you need is to merge the part of speech tags or entity labels onto the tokens before feeding the text to Gensim or another word2vec implementation. I've wanted to do this for a while, so it's nice to see that it works well.

danieldk 11y ago
Indeed, the implementation is very simple. We had considered this idea some times as well, but like you didn't get around to implement/evaluate it yet. So, it's good to hear that it works. One thing notably absent from the paper is a discussion of the trade-off between augmenting tokens with annotations in this way for sense disambiguation vs. data sparseness. Their approach may make the embeddings for frequent senses better, but the difficulty in WSD is typically in low-frequency senses. I think that particularly in disambiguation using part-of-speech tags, there is still a high semantic relatedness between senses, especially in languages with frequent nominalization or verbalization. I can imagine that an model that predicts a target (or context) in decomposed form (token and label) might improve embeddings for low-frequent senses.
- syllogism 11y ago
  This isn't really WSD though, or at least, only very weakly. Rare words are usually pretty unambiguous for part-of-speech. I would guess this mostly has an effect on the top 5,000 items of the vocabulary, and most of the rest of the lexicon only has a single "sense".
  - danieldk 11y ago
    This isn't really WSD though, or at least, only very weakly. Sure. I was pointing to real WSD where sparseness becomes even a stronger problem than when your definition of sense is restricted to part-of-speech tag or sentiment. Rare words are usually pretty unambiguous for part-of-speech. I was talking about (possibly) frequent words where some part-of-speech are infrequent, not about rare words. To take five more or less random examples form the Brown corpus (yes, we train on large corpora, but I think similar distributions could hold for less frequent forms in languages with e.g. frequent nomalization, not everyone speaks English!): mother NN 173 VB 1 code NN 20 VB 1 hanging NN 1 VBG 20 level JJ 14 NN 172 VB 2 services NNS 115 VBZ 1 If your learning method is as coarse-grained as simply throwing the token plus part-of-speech into word2vec or wang2vec, some will be below the frequency cut-offs (or will be to sparse to learn good embeddings), while other 'senses' may in reality be semantically similar.
    - syllogism 11y ago
      Thanks for the explanation. I see what you're saying now.
benten10 11y ago
This is really excellent for researchers approaching the field from the more NLP angle than the ML/CompLing angle. It was only a few weeks(days?) ago that Chris Manning came up with the whitepaper arguing that linguists have just as much/even more relevance now and shouldn't worry that neural networks will eat their lunch, and then we get to see this. This is awesome! : )
- YeGoblynQueenne 11y ago
  Have you a link to that whitepaper? I found an uncorrected draft, but I'd like to read the finished thing- cheers.
  - williamtrask 11y ago
    i think this one: http://www.mitpressjournals.org/doi/pdf/10.1162/COLI_a_00239 http://www.mitpressjournals.org/doi/pdf/10.1162/COLI_a_00239
    - YeGoblynQueenne 11y ago
      That's the draft one but thank you anyway :)

williamtrask 11y ago

working on it. we'll see.