4 ms·

As pre-built word vectors go, Conceptnet Numberbatch [1], introduced less flippantly as the ConceptNet Vector Ensemble [2], already outperforms this on all the

by rspeer 10y ago

As pre-built word vectors go, Conceptnet Numberbatch [1], introduced less flippantly as the ConceptNet Vector Ensemble [2], already outperforms this on all the measures evaluated in its paper: Rare Words, MEN-3000, and WordSim-353. This fact is hard to publicize because somehow the luminaries of the field decided that they didn't care about these evaluations anymore, back when RW performance was around 0.4. I have had reviewers dismiss it as "incremental improvements" to improve Rare Words from 0.4 to 0.6 and to improve MEN-3000 to be as good as a high estimate of inter-annotator agreement. It is possible to do much, much better than Google News skip-grams ("word2vec"), and one thing that helps get there is lexical knowledge of the kind that's in ConceptNet. [1] https://blog.conceptnet.io/2016/05/25/conceptnet-numberbatch-a-new-name-for-the-best-word-embeddings-you-can-download/ https://blog.conceptnet.io/2016/05/25/conceptnet-numberbatch... [2] https://blog.luminoso.com/2016/04/06/an-introduction-to-the-conceptnet-vector-ensemble/ https://blog.luminoso.com/2016/04/06/an-introduction-to-the-...

rspeer 10y ago
That said: LexVec gives quite good results on word-relatedness for using only distributional knowledge, and only from Wikipedia at that. Adding ConceptNet might give something that is more likely to be state-of-the-art.
- glup 10y ago
  ...And just distributional knowledge makes it easy to train new models on domain-specific corpora, or new languages. Is it possible to do the same with ConceptNet? I generally find that expert-derived ontologies suffer from bad coverage of low frequency items, rigidly discrete relationships, and are usually limited to a single language. That said, they're vastly better than nothing for a lot of tasks (same goes for WordNet).
  - rspeer 10y ago
    You can retrain your distributional knowledge and keep your lexical knowledge. Moving to a new domain shouldn't mean you have to forget everything about what words mean and hope you manage to learn it again. The whole idea of Numberbatch is that a combination of distributional and lexical knowledge is much better than either one alone. BTW, ConceptNet is only partially expert-derived (much of it is crowd-sourced), aims not to be rigid like WordNet is, and is in a whole lot of languages. "Retraining" ConceptNet itself is a bit of a chore, but you can do it. That is, you can get the source [1], add or remove sources of data, and rebuild it. Meanwhile, if you wanted to retrain word2vec's Google News skip-gram vectors, you would have to get a machine learning job at Google. [1] https://github.com/commonsense/conceptnet5 https://github.com/commonsense/conceptnet5
Karrot_Kream 10y ago
Thanks for bringing these tools to my attention! Awesome stuff!
argonaut 10y ago
Not to be too harsh, but if all you did was ensemble several different embeddings using methods that were mostly from other papers, it would be pretty obvious you'd get state of the art performance. But that is not very interesting from a novel theoretical, scientific, or even engineering perspective. It is certainly useful to the community, just not academic. Most people would consider it obvious. I could achieve state of the art performance on ImageNet if I created a larger ensemble of Inception v3 networks than the ensemble the paper used and fiddled around with different data augmentation tricks they were too lazy to use, but that's not very interesting.
- rspeer 10y ago
  Your claim of "I could do that" rings false. You didn't do it. You think that you could have done it because I wrote a clear enough paper telling you how. And even if you were curious and determined enough to do it, the thing that would make it possible for you to do it are the years that I have spent making ConceptNet. And when the "academic" things consistently do 10% or more worse than the "obvious" thing, maybe it's not that obvious.
  - argonaut 10y ago
    I was referring to ImageNet as to what I could do (because my experience is in computer vision). But more broadly, ensembling in general is really really obvious. I mean, for heavens sake, it's the trick everyone uses on Kaggle. You will find that is a very common reason why academics dismiss work for being too incremental - it is too obvious. It doesn't matter how many years you spent working on it. Whether or not your work is novel depends on what you did, not strictly on whether you managed to squeeze out more performance on a benchmark.
    - rspeer 10y ago
      Ensemble methods have been around for decades because they are a good idea. Watson -- the Jeopardy bot, not the marketing brand -- was an ensemble method. My impression is that people avoid ensembles because they are fighting to give the impression that their work is the only work that matters. It's "not invented here" syndrome. I'm glad I'm not trying to get tenure.
plusepsilon 10y ago
Cool stuff! It wasn't described in the paper (afaict), but did you have ConceptNet embeddings as well? I can certainly see a way of creating embeddings by using all the links as context. (e.g. http://conceptnet5.media.mit.edu/web/c/en/knowledge http://conceptnet5.media.mit.edu/web/c/en/knowledge) edit: actually looks like you "retrofit" existing word embeddings by re-weighing based on strengths of the links.
- rspeer 10y ago
  There kind of are ConceptNet embeddings. They weren't called "embeddings" when they were released, because it was 2008 [1]. Here's an example: http://conceptnet5.media.mit.edu/data/5.4/assoc/c/en/knowledge http://conceptnet5.media.mit.edu/data/5.4/assoc/c/en/knowled... We dropped working on those embeddings for a while because retrofitting was more effective. But we may have found a way to use the ConceptNet embeddings again, to improve languages besides English beyond what retrofitting is doing. [1] http://dspace.mit.edu/handle/1721.1/51870 http://dspace.mit.edu/handle/1721.1/51870

by rspeer 10y ago

rspeer 10y ago

That said: LexVec gives quite good results on word-relatedness for using only distributional knowledge, and only from Wikipedia at that. Adding ConceptNet might give something that is more likely to be state-of-the-art.

glup 10y ago
...And just distributional knowledge makes it easy to train new models on domain-specific corpora, or new languages. Is it possible to do the same with ConceptNet? I generally find that expert-derived ontologies suffer from bad coverage of low frequency items, rigidly discrete relationships, and are usually limited to a single language. That said, they're vastly better than nothing for a lot of tasks (same goes for WordNet).
- rspeer 10y ago
  You can retrain your distributional knowledge and keep your lexical knowledge. Moving to a new domain shouldn't mean you have to forget everything about what words mean and hope you manage to learn it again. The whole idea of Numberbatch is that a combination of distributional and lexical knowledge is much better than either one alone. BTW, ConceptNet is only partially expert-derived (much of it is crowd-sourced), aims not to be rigid like WordNet is, and is in a whole lot of languages. "Retraining" ConceptNet itself is a bit of a chore, but you can do it. That is, you can get the source [1], add or remove sources of data, and rebuild it. Meanwhile, if you wanted to retrain word2vec's Google News skip-gram vectors, you would have to get a machine learning job at Google. [1] https://github.com/commonsense/conceptnet5 https://github.com/commonsense/conceptnet5

Karrot_Kream 10y ago

Thanks for bringing these tools to my attention! Awesome stuff!

argonaut 10y ago

Not to be too harsh, but if all you did was ensemble several different embeddings using methods that were mostly from other papers, it would be pretty obvious you'd get state of the art performance. But that is not very interesting from a novel theoretical, scientific, or even engineering perspective. It is certainly useful to the community, just not academic. Most people would consider it obvious. I could achieve state of the art performance on ImageNet if I created a larger ensemble of Inception v3 networks than the ensemble the paper used and fiddled around with different data augmentation tricks they were too lazy to use, but that's not very interesting.

rspeer 10y ago
Your claim of "I could do that" rings false. You didn't do it. You think that you could have done it because I wrote a clear enough paper telling you how. And even if you were curious and determined enough to do it, the thing that would make it possible for you to do it are the years that I have spent making ConceptNet. And when the "academic" things consistently do 10% or more worse than the "obvious" thing, maybe it's not that obvious.
- argonaut 10y ago
  I was referring to ImageNet as to what I could do (because my experience is in computer vision). But more broadly, ensembling in general is really really obvious. I mean, for heavens sake, it's the trick everyone uses on Kaggle. You will find that is a very common reason why academics dismiss work for being too incremental - it is too obvious. It doesn't matter how many years you spent working on it. Whether or not your work is novel depends on what you did, not strictly on whether you managed to squeeze out more performance on a benchmark.
  - rspeer 10y ago
    Ensemble methods have been around for decades because they are a good idea. Watson -- the Jeopardy bot, not the marketing brand -- was an ensemble method. My impression is that people avoid ensembles because they are fighting to give the impression that their work is the only work that matters. It's "not invented here" syndrome. I'm glad I'm not trying to get tenure.

plusepsilon 10y ago

Cool stuff! It wasn't described in the paper (afaict), but did you have ConceptNet embeddings as well? I can certainly see a way of creating embeddings by using all the links as context. (e.g. http://conceptnet5.media.mit.edu/web/c/en/knowledge http://conceptnet5.media.mit.edu/web/c/en/knowledge) edit: actually looks like you "retrofit" existing word embeddings by re-weighing based on strengths of the links.

rspeer 10y ago
There kind of are ConceptNet embeddings. They weren't called "embeddings" when they were released, because it was 2008 [1]. Here's an example: http://conceptnet5.media.mit.edu/data/5.4/assoc/c/en/knowledge http://conceptnet5.media.mit.edu/data/5.4/assoc/c/en/knowled... We dropped working on those embeddings for a while because retrofitting was more effective. But we may have found a way to use the ConceptNet embeddings again, to improve languages besides English beyond what retrofitting is doing. [1] http://dspace.mit.edu/handle/1721.1/51870 http://dspace.mit.edu/handle/1721.1/51870