4 ms·

I'm not in academia, but in the grand tradition of "why don't you just..." solutions crossed with "technical solutions to people problems": Would it help at al

by sfink 5y ago

I'm not in academia, but in the grand tradition of "why don't you just..." solutions crossed with "technical solutions to people problems": Would it help at all if rather than participants reviewing 3 papers, each reviewed 2 papers and validated the review of 3 more papers? This is computer science here, with things like the set NP whose defining characteristic is that it's easier to check a solution than generate it. I'm imagining having some standard that reviews are held to in order to make them validatable. When validating a review, you are just confirming that the issues brought up are reasonable. Same for the compliments. Sure, it's not perfect because the validators wouldn't dive in as deep or have as much context as the reviewers, but sitting here in my obsidian tower of industry, it seems like it would at least make collusion attacks more difficult. Hopefully without increasing the already heavy load on reviewers. (It very much seems like an incomplete solution -- we only have to look at politics and regulatory capture to see how far wrong things can go, in ways immune to straightforward interventions. Really, you need to tear down as many of the obstacles to a culture of trust as you can. Taping over the holes in a leaking bucket doesn't work for long.)

snowwrestler 5y ago
That would only work if the review decisions could be expected to be reasonably consistent from person to person. But, from the article: > In a well-publicized case in 2014, organizers of the Neural Information Processing Systems Conference formed two independent program committees and had 10% of submissions reviewed by both. The result was that almost 60% of papers accepted by one program committee were rejected by the other, suggesting that the fate of many papers is determined by the specifics of the reviewers selected and not just the inherent value of the work itself. With this much demonstrated discrepancy between two sets of reviewers, it’s hard to believe that adding a validation step would produce a consistent improvement. How can people be expected to find improperly accepted papers when they have less than 50% agreement on the acceptance of good-faith submissions? Honestly I think this seeming randomness in acceptance is at the heart of why people might think cheating is acceptable. If the process is not reliable, why bother submitting to it?
- sjg007 5y ago
  >The result was that almost 60% of papers accepted by one program committee were rejected by the other, suggesting that the fate of many papers is determined by the specifics of the reviewers selected and not just the inherent value of the work itself. I referenced Kahneham's latest book, Noise, above but this is exactly the problem he focuses on. There are solutions.
  - cortesoft 5y ago
    Wouldn’t a ‘solution’ only work if there actually is some underlying objective quality to a paper? It might be the case that different reviewers disagree because there is no right answer, because there is no such thing as objective quality.
    - sjg007 5y ago
      Well wisdom of the crowds suggest that there is a truth... and a conference is charged with representing the best so I think that randomization and meta moderation + other strategies are better in the long run.
- btilly 5y ago
  The described result is not in conflict with decisions being reasonably consistent. Suppose that two reviewers independently rank papers 80% on quality and 20% on chance factors. With good odds, the two reviewers will agree with each other on the relative rankings of any given pair of papers. But their lists of the top 10% of papers will largely not be in agreement with each other.
- visarga 5y ago
  This is in aggregate. If you look at great papers, they are almost surely accepted. Bad papers are almost surely rejected. But those in the middle are a coin toss.
rsfern 5y ago
I think this is a really interesting idea. It does seem more effective for validating that reviewers are not torching the work of others — is there a nice way to validate that reviewers are not giving their friends a pass, short of reviewing the manuscript and the review?
lumost 5y ago
The problem with Academic collusion rings is that eventually the ring can hold such influence that all "major" research comes from the ring. As ring members benefit there is no reason for them to switch to an alternate system. If one wants to progress in their chosen field then there is no benefit to publishing in an unread source.

by sfink 5y ago

snowwrestler 5y ago

That would only work if the review decisions could be expected to be reasonably consistent from person to person. But, from the article: > In a well-publicized case in 2014, organizers of the Neural Information Processing Systems Conference formed two independent program committees and had 10% of submissions reviewed by both. The result was that almost 60% of papers accepted by one program committee were rejected by the other, suggesting that the fate of many papers is determined by the specifics of the reviewers selected and not just the inherent value of the work itself. With this much demonstrated discrepancy between two sets of reviewers, it’s hard to believe that adding a validation step would produce a consistent improvement. How can people be expected to find improperly accepted papers when they have less than 50% agreement on the acceptance of good-faith submissions? Honestly I think this seeming randomness in acceptance is at the heart of why people might think cheating is acceptable. If the process is not reliable, why bother submitting to it?

sjg007 5y ago
>The result was that almost 60% of papers accepted by one program committee were rejected by the other, suggesting that the fate of many papers is determined by the specifics of the reviewers selected and not just the inherent value of the work itself. I referenced Kahneham's latest book, Noise, above but this is exactly the problem he focuses on. There are solutions.
- cortesoft 5y ago
  Wouldn’t a ‘solution’ only work if there actually is some underlying objective quality to a paper? It might be the case that different reviewers disagree because there is no right answer, because there is no such thing as objective quality.
  - sjg007 5y ago
    Well wisdom of the crowds suggest that there is a truth... and a conference is charged with representing the best so I think that randomization and meta moderation + other strategies are better in the long run.
btilly 5y ago
The described result is not in conflict with decisions being reasonably consistent. Suppose that two reviewers independently rank papers 80% on quality and 20% on chance factors. With good odds, the two reviewers will agree with each other on the relative rankings of any given pair of papers. But their lists of the top 10% of papers will largely not be in agreement with each other.
visarga 5y ago
This is in aggregate. If you look at great papers, they are almost surely accepted. Bad papers are almost surely rejected. But those in the middle are a coin toss.

rsfern 5y ago

I think this is a really interesting idea. It does seem more effective for validating that reviewers are not torching the work of others — is there a nice way to validate that reviewers are not giving their friends a pass, short of reviewing the manuscript and the review?

lumost 5y ago

The problem with Academic collusion rings is that eventually the ring can hold such influence that all "major" research comes from the ring. As ring members benefit there is no reason for them to switch to an alternate system. If one wants to progress in their chosen field then there is no benefit to publishing in an unread source.