3 ms·
Sure thing! For the GIN-ε (https://arxiv.org/pdf/1810.00826.pdf https://arxiv.org/pdf/1810.00826.pdf), for example, the authors report a classification accurac
by Topolomancer 6y ago
Sure thing!
For the GIN-ε (https://arxiv.org/pdf/1810.00826.pdf https://arxiv.org/pdf/1810.00826.pdf), for example, the authors report a classification accuracy of 75.9±3.8 on the PROTEINS data set (classical graph benchmark data set). If you run it with a cross-validation setup that is repeated to account for effects of chance, performance drops to 73.1±0.7.
Notice the drop—the second accuracy value is at least within the standard deviation of the first one, but you can see that a different experimental setup shrinks the gains quite a lot...
Same goes for different data sets. Since the gains are not super large for most papers, these changes matter a lot. But of course, the paper is now published, so no one is going to go back and change it.
FWIW: I like the GIN paper and think the authors did a good job. It's just that their experimental setup is insufficiently thorough for the data sets they are considering, thus leading to overoptimistic estimates. This is a problem because the next 'state of the art' paper has to find a way to get a slightly higher mean accuracy, at the expense of an even larger standard deviation, etc.
- YeGoblynQueenne 6y agoThanks - it will take a while to read through the paper. I'm very surprised to see actual theoretical contributions in a (recent) neural networks paper. Pleasantly surprised. I agree the trend you point out is worrying. I suppose if this continues at some point the benchmarks are beaten, but that still tells us nothing about the true abilities of the tested systems or algorithms.
- Topolomancer 6y agoYou are welcome! If you want to go further down the rabbit hole, feel free to ping me via another communication channel; I have some interesting findings to share but they are unfortunately not ready yet for public consumption. (this sounds more ominous than I intended it to sound; the reason is plain and simple that the publication is still under review and we have no preprint)
- YeGoblynQueenne 6y agoHaha, don't worry, I thought you meant it's a draft or under review :) I got a research interest in GNNs and the datasets used in papers like the one you link, but I have so far only dipped a toe- because I have other priorities right now. But, more if I ping you :)