3 ms·

Peer Review is as bad as the Peers Reviewing. That’s often awful

3 points by barry-cotter 6y ago

barry-cotter 6y ago
> Peer review is fine for what it is—it tells you that a paper is up to standard in its subfield. Peer reviewers can catch missing references in the literature review. That can be helpful! But if peer review catches anything that the original authors didn’t understand . . . well, that’s just lucky. You certainly can’t expect it. ... > 4. All the papers that don’t get retracted . . . > One problem with the attention given to the fatally flawed papers that get retracted is that we forget all the fatally flawed papers that aren’t retracted. > For example, from that long paragraph above: > – The beauty-and-sex-ratio paper: Not retracted. 58 citations. Endorsement from Freakonomics never rescinded. > – The ovulation-and-voting paper: Not retracted. 107 citations. Fortunately this one never got taken seriously by the news media. > – The gremlins paper. Not retracted: 1064 citations. Might still be influencing some policy debates. Etc etc etc. > And, if we just want to look at papers published in Lancet: > – That seriously flawed Iraq survey: Never retracted. 779 citations. Came up in policy debates. > – That hopeless-from-the-start paper on gun control based on an unregularized regression with 50 data points, 25+ predictors, and a bunch of ridiculous conclusions: OK, this one had only 75 citations and, fortunately, it was blasted in the press when it came out. So, too bad that something like 75 people thought this paper was worth citing (and, no, a quick glance at the citations suggests that these are not 75 papers using this as an example to show people how not to do policy analysis). As I wrote the other day, the useless study is included in a meta-analysis published in JAMA—and one of the authors of that meta-analysis is the person who said he did not believe the Lancet paper when it came out! But now it’s in the literature now and it’s not going away.

3 points by barry-cotter 6y ago

> Peer review is fine for what it is—it tells you that a paper is up to standard in its subfield. Peer reviewers can catch missing references in the literature review. That can be helpful! But if peer review catches anything that the original authors didn’t understand . . . well, that’s just lucky. You certainly can’t expect it. ... > 4. All the papers that don’t get retracted . . . > One problem with the attention given to the fatally flawed papers that get retracted is that we forget all the fatally flawed papers that aren’t retracted. > For example, from that long paragraph above: > – The beauty-and-sex-ratio paper: Not retracted. 58 citations. Endorsement from Freakonomics never rescinded. > – The ovulation-and-voting paper: Not retracted. 107 citations. Fortunately this one never got taken seriously by the news media. > – The gremlins paper. Not retracted: 1064 citations. Might still be influencing some policy debates. Etc etc etc. > And, if we just want to look at papers published in Lancet: > – That seriously flawed Iraq survey: Never retracted. 779 citations. Came up in policy debates. > – That hopeless-from-the-start paper on gun control based on an unregularized regression with 50 data points, 25+ predictors, and a bunch of ridiculous conclusions: OK, this one had only 75 citations and, fortunately, it was blasted in the press when it came out. So, too bad that something like 75 people thought this paper was worth citing (and, no, a quick glance at the citations suggests that these are not 75 papers using this as an example to show people how not to do policy analysis). As I wrote the other day, the useless study is included in a meta-analysis published in JAMA—and one of the authors of that meta-analysis is the person who said he did not believe the Lancet paper when it came out! But now it’s in the literature now and it’s not going away.