3 ms·

> the author notes that it is well-known that the quality of the work has little to do with its odds of acceptance even when the system is 'functioning' as des

> the author notes that it is well-known that the quality of the work has little to do with its odds of acceptance even when the system is 'functioning' as designed I think this is a little more pessimistic than what the piece says. The NeurIPS (then-NIPS) experiment said that about 60% of papers accepted by one PC got rejected by the other. That doesn't actually mean "the quality of the work has little to do with its odds of acceptance". It may just be that there's a paper has to cross a quality threshold, and once it's past, then the outcome has a lot more variation. My personal take on NeurIPS specifically is that there's a fraction of bad papers, maybe 40%, that probably shouldn't and won't get in. Then there's a minority of very nice papers that probably should and will get in, maybe 5-10%. And then there are a bunch of middling papers where a lot of it is luck and drawing friendly reviewers. But these aren't bad papers, and you can't really just churn them out, they're just not very good papers.

fulafel 5y ago
Seems it's also rational for an opportunistic player of the publications game to split their work into the maximal number of marginally acceptable publications, which would result in a high rejection %.
- xmprt 5y ago
  I wouldn't be surprised if that's already happening and has become the preferred strategy for the last decade at least.
  - Teever 5y ago
    https://en.wikipedia.org/wiki/Least_publishable_unit https://en.wikipedia.org/wiki/Least_publishable_unit
wiz21c 5y ago
then why not review projects in a double blind fashion ? Those which are accepted twice will get published and voilà. But that's twice the work.
- mycologos 5y ago
  Double-blind usually means something else in reviewing: the reviewer does not know who the author of the paper is, and the author does not learn who the reviewer is. Single-blind reviewing is when the reviewer knows who the author of the paper is, but the author still does not learn who the reviewer is. As you mention, the problem with having two sets of reviewers is that it's hard enough for conferences like NeurIPS to find one set of qualified reviewers. Usually at least 1/3 of reviewers on any given paper produce a poor review, either because they don't care or because they really lack expertise. Complaining about this is so widespread that it even doubles as a sort of icebreaker for researchers, but nobody has a good solution.
Beldin 5y ago
Don't forget there can be papers recognised universally as acceptable. From memory, it was 25% rejected by both PCs, 15% accepted by both PCs, and the middle 60% random.

fulafel 5y ago

Seems it's also rational for an opportunistic player of the publications game to split their work into the maximal number of marginally acceptable publications, which would result in a high rejection %.

xmprt 5y ago
I wouldn't be surprised if that's already happening and has become the preferred strategy for the last decade at least.
- Teever 5y ago
  https://en.wikipedia.org/wiki/Least_publishable_unit https://en.wikipedia.org/wiki/Least_publishable_unit

wiz21c 5y ago

then why not review projects in a double blind fashion ? Those which are accepted twice will get published and voilà. But that's twice the work.

mycologos 5y ago
Double-blind usually means something else in reviewing: the reviewer does not know who the author of the paper is, and the author does not learn who the reviewer is. Single-blind reviewing is when the reviewer knows who the author of the paper is, but the author still does not learn who the reviewer is. As you mention, the problem with having two sets of reviewers is that it's hard enough for conferences like NeurIPS to find one set of qualified reviewers. Usually at least 1/3 of reviewers on any given paper produce a poor review, either because they don't care or because they really lack expertise. Complaining about this is so widespread that it even doubles as a sort of icebreaker for researchers, but nobody has a good solution.

Beldin 5y ago

Don't forget there can be papers recognised universally as acceptable. From memory, it was 25% rejected by both PCs, 15% accepted by both PCs, and the middle 60% random.