4 ms·

I was referring to ImageNet as to what I could do (because my experience is in computer vision). But more broadly, ensembling in general is really really obviou

I was referring to ImageNet as to what I could do (because my experience is in computer vision). But more broadly, ensembling in general is really really obvious. I mean, for heavens sake, it's the trick everyone uses on Kaggle. You will find that is a very common reason why academics dismiss work for being too incremental - it is too obvious. It doesn't matter how many years you spent working on it. Whether or not your work is novel depends on what you did, not strictly on whether you managed to squeeze out more performance on a benchmark.

rspeer 10y ago
Ensemble methods have been around for decades because they are a good idea. Watson -- the Jeopardy bot, not the marketing brand -- was an ensemble method. My impression is that people avoid ensembles because they are fighting to give the impression that their work is the only work that matters. It's "not invented here" syndrome. I'm glad I'm not trying to get tenure.
- argonaut 10y ago
  People avoid ensembles, unless they're trying to squeeze out higher performance on a benchmark (or they're doing research on the technique of ensembling in of itself). I would have thought this would be obvious to you, since you're a researcher.
  - rspeer 10y ago
    My company, Luminoso, uses Conceptnet Numberbatch as one component for interpreting the meaning of text. When the benchmarks went up, the understandability of its results did too. You get better search results, better topics, clearer visualizations. I'm not just trying to squeeze out extra performance, I'm trying to make computers understand text better. The benchmarks are the evidence that it's better. I do consider myself a researcher despite leaving academia, and having some respect for evidence is part of that identity. When academia decides to disregard evidence because evidence is for stupid Kagglers (I don't use Kaggle but I respect a good result when I see one), that's how they end up lagging behind open source software. I understand that it's not worthwhile to chase every evaluation. For example, some evaluations are unrepresentative. Some evaluations, like parsing according to the Penn Treebank, were once useful but have been squeezed dry in a way that doesn't reflect real-world performance. And some tasks chase these unhelpful evaluations. But I would credit Kaggle with making academics realize, slowly, that they should use random forests as a baseline when evaluating a classification method. People were content to publish classifiers that were worse than random forests until Kaggle presented overwhelming evidence that random forests worked better than many techniques. In text understanding, the fact that seems not to have taken hold -- one that I think should be obvious, even -- is "computers understand text better when they know more facts about words". This is what ConceptNet (not the whole ensemble, but ConceptNet itself) is about.
    - argonaut 10y ago
      This is all good. It justifies a technical report, a blog post, a workshop paper, or publication in venues looking for this kind of owrk. It doesn't necessarily justify publication in a venue looking for totally new ideas.

rspeer 10y ago

Ensemble methods have been around for decades because they are a good idea. Watson -- the Jeopardy bot, not the marketing brand -- was an ensemble method. My impression is that people avoid ensembles because they are fighting to give the impression that their work is the only work that matters. It's "not invented here" syndrome. I'm glad I'm not trying to get tenure.

argonaut 10y ago
People avoid ensembles, unless they're trying to squeeze out higher performance on a benchmark (or they're doing research on the technique of ensembling in of itself). I would have thought this would be obvious to you, since you're a researcher.
- rspeer 10y ago
  My company, Luminoso, uses Conceptnet Numberbatch as one component for interpreting the meaning of text. When the benchmarks went up, the understandability of its results did too. You get better search results, better topics, clearer visualizations. I'm not just trying to squeeze out extra performance, I'm trying to make computers understand text better. The benchmarks are the evidence that it's better. I do consider myself a researcher despite leaving academia, and having some respect for evidence is part of that identity. When academia decides to disregard evidence because evidence is for stupid Kagglers (I don't use Kaggle but I respect a good result when I see one), that's how they end up lagging behind open source software. I understand that it's not worthwhile to chase every evaluation. For example, some evaluations are unrepresentative. Some evaluations, like parsing according to the Penn Treebank, were once useful but have been squeezed dry in a way that doesn't reflect real-world performance. And some tasks chase these unhelpful evaluations. But I would credit Kaggle with making academics realize, slowly, that they should use random forests as a baseline when evaluating a classification method. People were content to publish classifiers that were worse than random forests until Kaggle presented overwhelming evidence that random forests worked better than many techniques. In text understanding, the fact that seems not to have taken hold -- one that I think should be obvious, even -- is "computers understand text better when they know more facts about words". This is what ConceptNet (not the whole ensemble, but ConceptNet itself) is about.
  - argonaut 10y ago
    This is all good. It justifies a technical report, a blog post, a workshop paper, or publication in venues looking for this kind of owrk. It doesn't necessarily justify publication in a venue looking for totally new ideas.