4 ms·

PeerReview4All: Fair and accurate reviewer assignment in peer review

47 points by privong 3y ago

sideshowb 3y ago
On arxiv, of all places ;-)
- v-yadli 3y ago
  To be fair though, it's not like the other places have a suitable ``meta track'' to discuss such issues...
- mrdrozdov 3y ago
  This paper has been online at least since 2018. It has since been published: https://proceedings.mlr.press/v98/stelmakh19a.html https://proceedings.mlr.press/v98/stelmakh19a.html
alphazard 3y ago
It's probably time to sunset peer review. It currently functions to keep outsiders and dissidents from publishing in a field. I would rather see some sort of prediction market where scientists make claims betting against one another, and 3rd parties are incentivized to conduct experiments based on the total value at stake.
- anon__coward 3y ago
  I am curious: what is your background and experience? Please don’t take this as hostile as it’s not intended to be, I am honestly curious.
  - alpineidyll3 3y ago
    As a former faculty member who now directs science in industry. Agree.
- yarpen_z 3y ago
  No offense, but your solution reads like a typical idea that Sillicon Valley startup could come up with, i.e., throwing technology at a wrong problem, ignoring human factor and applying monetization where it doesn't belong. It reminds me of this founder who wanted to replace hiring processes with people putting money on the person staying in the firm long enough. Not all science is based on experiments. Furthermore, science is not usually done in two simple steps: creating a theory and validating it. In practice, it's a cycle of refirenement that often requires dozens of iterations and involves discovering new phenomena. In your model, who decides if the 3rd party has invalidated the claim or simply conducted the experiment incorrectly? Everyone involved has a financial motivation, so there's no independent side to resolve this issue.
  - patel011393 3y ago
    Perfectly said. Before attempting to solve a problem, it's important to consider the lenses/perspectives and that facts that you'll see nails everywhere with your hammers. Peer review can be more structured, is already blinded in many cases, and could benefit from regular crowdsourced suggestions from a variety of scholars. In many cases, journals have begun many reform practices like valuing open sharing of materials and code, structured transparency checklists, and opportunities to publish null results. All wonderful advances that don't require hammers.
  - alphazard 3y ago
    If we had a good way to pay people based on the value they add (basically solved for sales, but not for any other role), then you could construct a hiring process by betting on that value, and the most predictive interviewers would be rewarded. But until that's solved, I don't think there is a foundation for the betting. Time at the firm is the wrong metric. > Not all science is based on experiments. Okay, well I'm not talking about that kind of science. I'm talking about the science where the value of a theory is measured in it's ability to make testable predictions. I'm not really sure how the the other kind of scientists resolve disagreements. I didn't claim to have solved the problem. I was outlining what I think a solution would look like, and contrasting that to the idea of peer review as the way to determine what is scientifically relevant, if we could only make it more fair.

47 points by privong 3y ago

sideshowb 3y ago

On arxiv, of all places ;-)

v-yadli 3y ago
To be fair though, it's not like the other places have a suitable ``meta track'' to discuss such issues...
mrdrozdov 3y ago
This paper has been online at least since 2018. It has since been published: https://proceedings.mlr.press/v98/stelmakh19a.html https://proceedings.mlr.press/v98/stelmakh19a.html

alphazard 3y ago

It's probably time to sunset peer review. It currently functions to keep outsiders and dissidents from publishing in a field. I would rather see some sort of prediction market where scientists make claims betting against one another, and 3rd parties are incentivized to conduct experiments based on the total value at stake.

anon__coward 3y ago
I am curious: what is your background and experience? Please don’t take this as hostile as it’s not intended to be, I am honestly curious.
- alpineidyll3 3y ago
  As a former faculty member who now directs science in industry. Agree.
yarpen_z 3y ago
No offense, but your solution reads like a typical idea that Sillicon Valley startup could come up with, i.e., throwing technology at a wrong problem, ignoring human factor and applying monetization where it doesn't belong. It reminds me of this founder who wanted to replace hiring processes with people putting money on the person staying in the firm long enough. Not all science is based on experiments. Furthermore, science is not usually done in two simple steps: creating a theory and validating it. In practice, it's a cycle of refirenement that often requires dozens of iterations and involves discovering new phenomena. In your model, who decides if the 3rd party has invalidated the claim or simply conducted the experiment incorrectly? Everyone involved has a financial motivation, so there's no independent side to resolve this issue.
- patel011393 3y ago
  Perfectly said. Before attempting to solve a problem, it's important to consider the lenses/perspectives and that facts that you'll see nails everywhere with your hammers. Peer review can be more structured, is already blinded in many cases, and could benefit from regular crowdsourced suggestions from a variety of scholars. In many cases, journals have begun many reform practices like valuing open sharing of materials and code, structured transparency checklists, and opportunities to publish null results. All wonderful advances that don't require hammers.
- alphazard 3y ago
  If we had a good way to pay people based on the value they add (basically solved for sales, but not for any other role), then you could construct a hiring process by betting on that value, and the most predictive interviewers would be rewarded. But until that's solved, I don't think there is a foundation for the betting. Time at the firm is the wrong metric. > Not all science is based on experiments. Okay, well I'm not talking about that kind of science. I'm talking about the science where the value of a theory is measured in it's ability to make testable predictions. I'm not really sure how the the other kind of scientists resolve disagreements. I didn't claim to have solved the problem. I was outlining what I think a solution would look like, and contrasting that to the idea of peer review as the way to determine what is scientifically relevant, if we could only make it more fair.