4 ms·

I understand this point, yet to correctly assess whether a typical paper in my field - for instance - can be trusted you would need a modern neural networks res

I understand this point, yet to correctly assess whether a typical paper in my field - for instance - can be trusted you would need a modern neural networks researcher, a statistician, a cognitive scientist, a neuroscientist and perhaps a physicist. All would have to know about common fallacies in their respective fields and should optimally discuss with another. Faculty reviewers are not only overworked, they have accumulated biases, can be overconfident about their expertise and in the worst case have malicious incentive to prohibit publication of your results. I’m certain that a team of specialists who review as their full time job would do a better job than at least half of the reviewers I encountered (especially those at high-profile AI conferences). I enjoy reviewing too and am doing my best to be constructive, but I’m not sufficiently qualified for most of the offers I receive.

michaelmior 6y ago
I suppose a lot of this is dependent on the field. I think anyone who has had their work peer reviewed has many stories of reviews from people who seem to have completely misunderstood the work in question. Certainly there can be accumulated biases and overconfidence among faculty reviewers, but I don't see how that would be less prevalent among paid reviewers.
rocqua 6y ago
Would a person / team with the qualifications you mention best spend his time on reviewing such work, or should they also be doing some research. My point is, the only way to get the necessary skillset for reviewing is to actually do research. Hence it is peer-review. Once you are doing scientific research, there generally isn't anyone left who is 'better at it then you' so the best you can do is have it be reviewed by people who are 'roughly as good at it as you'. To ensure the long-term accuracy, we need both attempts at reproduction, and critical reading by people post publication. Both of those should have the option to cause the article to be retracted.

michaelmior 6y ago

I suppose a lot of this is dependent on the field. I think anyone who has had their work peer reviewed has many stories of reviews from people who seem to have completely misunderstood the work in question. Certainly there can be accumulated biases and overconfidence among faculty reviewers, but I don't see how that would be less prevalent among paid reviewers.

rocqua 6y ago

Would a person / team with the qualifications you mention best spend his time on reviewing such work, or should they also be doing some research. My point is, the only way to get the necessary skillset for reviewing is to actually do research. Hence it is peer-review. Once you are doing scientific research, there generally isn't anyone left who is 'better at it then you' so the best you can do is have it be reviewed by people who are 'roughly as good at it as you'. To ensure the long-term accuracy, we need both attempts at reproduction, and critical reading by people post publication. Both of those should have the option to cause the article to be retracted.