5 ms·

How Did That Make It Through Peer Review?

13 points by iamjeff 10y ago

anbende 10y ago
The author tries to make a case that this is not about incompetence. One of the primary examples given is the case where shoddy analyses sail through peer review, because none of the reviewers know enough about those analyses to comment. It's hard for me not to see this as incompetent. Are the reviewers incompetent for not knowing this particular analysis well enough to review it? Often! Most scientists in my experience (graduate student) have a pretty bad understanding of statistics and perform those analyses that will "pass peer review" rather than those which come from a comprehensive knowledge of the topic. And even if the analysis is new or uncommon, giving the paper a pass without mentioning that the analysis is beyond them is pretty problematic but is sadly also the norm (usually they could learn enough to review it in a few hours if they really wanted to!). Then there's the more general fact that no one reviewing the paper is qualified to comment on what is arguably the most critical part - whether analyses actually show what the authors think they do! If no one involved knows how to do the analyses in question then the team is not competent to perform or review this work. That's the result of sometimes an incompetent editor and usually a horribly broken system. The fact that checking the numbers isn't of primary importance in reviewing a lot of scientific work is a travesty.
chris_wot 10y ago
Makes you wonder whether peer review is working if a true subject matter expert doesn't do the review. Sounds like pointless rubber stamping and checkbox ticking.
dekhn 10y ago
Journal of Cardiology maintains a full-time statistics editor who reviews every single paper before publication. They routinely catch basic stats errors and send the papers back with a number of repairs required for the paper to be published. That Nature and Science and other prominent journals continue to publish statistically invalid papers without verifying them (and this isn't even peer review, it's just technical validation) amazes me.

13 points by iamjeff 10y ago

The author tries to make a case that this is not about incompetence. One of the primary examples given is the case where shoddy analyses sail through peer review, because none of the reviewers know enough about those analyses to comment. It's hard for me not to see this as incompetent. Are the reviewers incompetent for not knowing this particular analysis well enough to review it? Often! Most scientists in my experience (graduate student) have a pretty bad understanding of statistics and perform those analyses that will "pass peer review" rather than those which come from a comprehensive knowledge of the topic. And even if the analysis is new or uncommon, giving the paper a pass without mentioning that the analysis is beyond them is pretty problematic but is sadly also the norm (usually they could learn enough to review it in a few hours if they really wanted to!). Then there's the more general fact that no one reviewing the paper is qualified to comment on what is arguably the most critical part - whether analyses actually show what the authors think they do! If no one involved knows how to do the analyses in question then the team is not competent to perform or review this work. That's the result of sometimes an incompetent editor and usually a horribly broken system. The fact that checking the numbers isn't of primary importance in reviewing a lot of scientific work is a travesty.

chris_wot 10y ago

Makes you wonder whether peer review is working if a true subject matter expert doesn't do the review. Sounds like pointless rubber stamping and checkbox ticking.

dekhn 10y ago

Journal of Cardiology maintains a full-time statistics editor who reviews every single paper before publication. They routinely catch basic stats errors and send the papers back with a number of repairs required for the paper to be published. That Nature and Science and other prominent journals continue to publish statistically invalid papers without verifying them (and this isn't even peer review, it's just technical validation) amazes me.