3 ms·

The main take-away and problem for me is the 'arXiv dilemma', combined with shoddy scholarship: newcomers to the field regularly try to drink from the arXiv fir

The main take-away and problem for me is the 'arXiv dilemma', combined with shoddy scholarship: newcomers to the field regularly try to drink from the arXiv firehose and take every paper they find on there as gospel—even though it is not peer-reviewed (let's set aside the issues of peer review for that one). The quick publication cycle creates an environment that is always just about 'beating' the state of the art, but if you look closer into the reported values, you will often find a lot of questionable experimental choices. In one of my main application areas, viz. graph classification, almost none of the papers holds up (with respected to the reported performance gains) if subjected to a thorough experimental setup. This creates a dangerous environment; in the worst case, we might miss some interesting contributions because they are drowned by the noise of reviewers (here we go again!) claiming that 'It does not beat the state of the art, so it must be crap'.

dekhn 6y ago
I think this is a main reason that journal clubs exist in grad school: to refine the ability of PhDs to separate the bullshit from the real. I was shocked at how easy it was to "take down" a paper based on straightforward errors in the methods or analysis.
YeGoblynQueenne 6y ago
>> In one of my main application areas, viz. graph classification, almost none of the papers holds up (with respected to the reported performance gains) if subjected to a thorough experimental setup. Could you give an example? Just being curious :)
- Topolomancer 6y ago
  Sure thing! For the GIN-ε (https://arxiv.org/pdf/1810.00826.pdf https://arxiv.org/pdf/1810.00826.pdf), for example, the authors report a classification accuracy of 75.9±3.8 on the PROTEINS data set (classical graph benchmark data set). If you run it with a cross-validation setup that is repeated to account for effects of chance, performance drops to 73.1±0.7. Notice the drop—the second accuracy value is at least within the standard deviation of the first one, but you can see that a different experimental setup shrinks the gains quite a lot... Same goes for different data sets. Since the gains are not super large for most papers, these changes matter a lot. But of course, the paper is now published, so no one is going to go back and change it. FWIW: I like the GIN paper and think the authors did a good job. It's just that their experimental setup is insufficiently thorough for the data sets they are considering, thus leading to overoptimistic estimates. This is a problem because the next 'state of the art' paper has to find a way to get a slightly higher mean accuracy, at the expense of an even larger standard deviation, etc.
  - YeGoblynQueenne 6y ago
    Thanks - it will take a while to read through the paper. I'm very surprised to see actual theoretical contributions in a (recent) neural networks paper. Pleasantly surprised. I agree the trend you point out is worrying. I suppose if this continues at some point the benchmarks are beaten, but that still tells us nothing about the true abilities of the tested systems or algorithms.
    - Topolomancer 6y ago
      You are welcome! If you want to go further down the rabbit hole, feel free to ping me via another communication channel; I have some interesting findings to share but they are unfortunately not ready yet for public consumption. (this sounds more ominous than I intended it to sound; the reason is plain and simple that the publication is still under review and we have no preprint)
      - YeGoblynQueenne 6y ago
        Haha, don't worry, I thought you meant it's a draft or under review :) I got a research interest in GNNs and the datasets used in papers like the one you link, but I have so far only dipped a toe- because I have other priorities right now. But, more if I ping you :)

dekhn 6y ago

I think this is a main reason that journal clubs exist in grad school: to refine the ability of PhDs to separate the bullshit from the real. I was shocked at how easy it was to "take down" a paper based on straightforward errors in the methods or analysis.

YeGoblynQueenne 6y ago

>> In one of my main application areas, viz. graph classification, almost none of the papers holds up (with respected to the reported performance gains) if subjected to a thorough experimental setup. Could you give an example? Just being curious :)

Topolomancer 6y ago
Sure thing! For the GIN-ε (https://arxiv.org/pdf/1810.00826.pdf https://arxiv.org/pdf/1810.00826.pdf), for example, the authors report a classification accuracy of 75.9±3.8 on the PROTEINS data set (classical graph benchmark data set). If you run it with a cross-validation setup that is repeated to account for effects of chance, performance drops to 73.1±0.7. Notice the drop—the second accuracy value is at least within the standard deviation of the first one, but you can see that a different experimental setup shrinks the gains quite a lot... Same goes for different data sets. Since the gains are not super large for most papers, these changes matter a lot. But of course, the paper is now published, so no one is going to go back and change it. FWIW: I like the GIN paper and think the authors did a good job. It's just that their experimental setup is insufficiently thorough for the data sets they are considering, thus leading to overoptimistic estimates. This is a problem because the next 'state of the art' paper has to find a way to get a slightly higher mean accuracy, at the expense of an even larger standard deviation, etc.
- YeGoblynQueenne 6y ago
  Thanks - it will take a while to read through the paper. I'm very surprised to see actual theoretical contributions in a (recent) neural networks paper. Pleasantly surprised. I agree the trend you point out is worrying. I suppose if this continues at some point the benchmarks are beaten, but that still tells us nothing about the true abilities of the tested systems or algorithms.
  - Topolomancer 6y ago
    You are welcome! If you want to go further down the rabbit hole, feel free to ping me via another communication channel; I have some interesting findings to share but they are unfortunately not ready yet for public consumption. (this sounds more ominous than I intended it to sound; the reason is plain and simple that the publication is still under review and we have no preprint)
    - YeGoblynQueenne 6y ago
      Haha, don't worry, I thought you meant it's a draft or under review :) I got a research interest in GNNs and the datasets used in papers like the one you link, but I have so far only dipped a toe- because I have other priorities right now. But, more if I ping you :)