4 ms·

Eh, this isn't really surprising. The purpose of research is to explore- there are no guarantees that newer, complex methods will be better than simpler methods

• ska 7y ago
  > And I don't blame them- it could impact funding, access to journals, etc. No, being "understanding" about this makes you part of the problem. Part of research is trying new things, and exploring, but you should always be checking this against simpler techniques and currently accepted best performance. If it isn't an improvement, you don't publish at all unless you have something interesting to say about why it isn't improving things. Or perhaps in a methods survey. One thing that happens when a technique like deep learning gets popular and things like python toolkits pop up making it easy to try, is that you get researchers in adjacent areas who don't really understand what they are doing trying to apply it to their favorite problem domain, and then comparing almost exclusively to other naive groups doing the same thing. It can be hard to tell if there is anything interesting there, even when there are dozens of conference papers, etc. Basically the same thing happened with kernel methods when SVMs were hot, to a smaller scale. Compare this to say, AlexNet. The reason that was immediately obvious something interesting was happening there was the fact that lots of people who did know what they were doing in models, had tried lots of other approaches, and you could make direct comparisons. So yes, blame them. I do think negative results should be valued higher, but the fact you did some work doesn't make it publishable. Frame it another way, if you give me a paper proposing a complex model and I grab your data an play with it a bit and it turns out linear regression with appropriate pre treatment works better ... well then I'm forced to believe you are either incompetent or being dishonest. Or, if students, your supervisors are. This generalizes well. You should always be comparing against a known (or considered) good solution on your data under the same conditions, not just comparing against last years conference paper and variants you are trying to "improve". The right choice of baseline comparison will depend a bit on the domain, but not including it at all is shockingly poor scholarship. I've even seen paper submissions with no direct comparisons at all, because the "researchers" didn't have access to the comparators data, and were too lazy to implement the other methods. Which leads to another sloppiness - methods that get pulled into comparison not because they are the right choice, but because there is a publicly available implementation. In the best case this forms a useful baseline. In the worst case, well, I guess it's good for your citation count if you implemented it :)
  • SubiculumCode 7y ago
    Whether an approach is currently better than other well understood approaches shouldn't be the focus of researchers. It might be the focus of private sphere engineers, but researchers should be interested in whether a new approach has potential to advance machine learning.
    • logifail 7y ago
      > Whether an approach is currently better than other well understood approaches shouldn't be the focus of researchers Research (noun): "the systematic investigation into and study of materials and sources in order to establish facts and reach new conclusions." Q: How can one reach new conclusions without being aware of existing approaches and conclusions?
    • ska 7y ago
      Apologies; I think I was unclear. I don't mean to suggest that papers should boil down to performance comparisons with baseline results, not at all. What I'm saying is that if you don't do this comparison somewhere, it can be very hard to tell what your numeric results do mean. In the worse case you see people offering numerical comparisons to other approaches that are similarly unpinned, and you can't tell if they are interesting even if they are apples to apples. As a researcher, you are being lax if you never do that work if for nothing else than a sanity check on your implementation. If you've done it, it's good to include in the paper as a point of reference. If you currently aren't performing better than baseline, that's fine - but you should understand why and discuss that with insight too. So it isn't the focus. But it is table stakes that you understand this stuff.
  • sdinsn 7y ago
    > I do think negative results should be valued higher ...so, we are in agreement then?