6 ms·

> It’s also led to an undercurrent of anxiety among scientists who fear that their labs and their publication records might come under attack from a feisty cadr

> It’s also led to an undercurrent of anxiety among scientists who fear that their labs and their publication records might come under attack from a feisty cadre of freelance critics. Formerly known as "peer review".

lutusp 10y ago
> Formerly known as "peer review". An in-depth review of a paper in advance of publication is common (although that system isn't as reliable as had one been assumed). Such an exhaustive review after publication is rare, for multiple reasons. One serious problem with post-publication review is that the original data are rarely made available to reviewers. For example, academic critics of the work of Diederik Stapel couldn't justify revoking his Ph.D. because the original data that might have proved his fraud no longer existed. He eventually voluntarily gave up his degree[1], possibly to try to avoid a deeper investigation into his behavior. 1. http://www.science20.com/cool-links/dutch_psychologist_diederik_stapel_relinquishes_his_phd-84542 http://www.science20.com/cool-links/dutch_psychologist_diede...
- apathy 10y ago
  I love how you believe that most reviewers have time to conduct a thorough review for a journal, most of the time. As opposed to something much closer to "LGTM!" With some "but be sure to cite my vaguely-relevant work in Foo et al, 1952".
  - lutusp 10y ago
    > I love how you believe that most reviewers have time to conduct a thorough review for a journal, most of the time. Your objection is to a position I never took -- in fact I suggested the opposite. The depth of a review is gauged by its length, not its accuracy or objectivity.
    - apathy 10y ago
      Ah, OK. I think we are on the same page. My mistake.
Jweb_Guru 10y ago
Bluntly, no. Laypeople are not the intended audience for scientific papers and usually lack the background to provide an informed review (hell, even most researchers in the field do). They do lack certain biases and incentives that researchers familiar with the topic usually have, but that doesn't compensate for what they don't know (for example, whether the claimed phenomenon has been tested many other times and not shown up; whether the claimed mechanism is bunk; whether the supposedly novel idea has already been published many other times). That being said, laypeople can (and should) absolutely call out research malpractice like what's described in the article.
- wjnc 10y ago
  I disagree pretty strongly. You've got worldwide universities churning out masters and pdh-level educated persons, who mostly go to the private sector for employment. Many of those would be perfectly able to perform peer review. And with this p-hacking crisis, mostly due to abject statistical knowledge, there are literally millions of technically able non-scientists able to review this nonsense. Any physicist, econometricist, statistician (and >100 more technical fields) could kindly take a shot at the social science methods in dispute here.
  - Jweb_Guru 10y ago
    You're focusing on exactly one aspect of peer review, determining whether the statistical methods used were valid, which I agree can be done by pretty much anyone with a background in statistics (given that these days it's pretty easy to get in touch with the authors to ask for clarification). But peer review is about more than just determining whether the paper is internally coherent (or directly lifts text from other papers by the same author, which can be determined by a machine). I shouldn't have to say this, but a former physics Ph.D. is emphatically not qualified to perform peer review on papers on cell biology solely because physics and cell biology are both science. Papers don't exist in a vacuum, and by engaging in practices like selective citation (and others I mentioned in the previous post), it's possible to convince intelligent and reasonable people of a point that is completely wrong. In fact, it's so easy it even happens somewhat routinely within narrow subspecialties, where there may only be a handful of experts on a phenomenon worldwide. People who aren't up to date with the field already usually don't have time to spend weeks, months, or years trudging through the literature to gain enough background knowledge to make those kinds of decisions.

lutusp 10y ago

> Formerly known as "peer review". An in-depth review of a paper in advance of publication is common (although that system isn't as reliable as had one been assumed). Such an exhaustive review after publication is rare, for multiple reasons. One serious problem with post-publication review is that the original data are rarely made available to reviewers. For example, academic critics of the work of Diederik Stapel couldn't justify revoking his Ph.D. because the original data that might have proved his fraud no longer existed. He eventually voluntarily gave up his degree[1], possibly to try to avoid a deeper investigation into his behavior. 1. http://www.science20.com/cool-links/dutch_psychologist_diederik_stapel_relinquishes_his_phd-84542 http://www.science20.com/cool-links/dutch_psychologist_diede...

apathy 10y ago
I love how you believe that most reviewers have time to conduct a thorough review for a journal, most of the time. As opposed to something much closer to "LGTM!" With some "but be sure to cite my vaguely-relevant work in Foo et al, 1952".
- lutusp 10y ago
  > I love how you believe that most reviewers have time to conduct a thorough review for a journal, most of the time. Your objection is to a position I never took -- in fact I suggested the opposite. The depth of a review is gauged by its length, not its accuracy or objectivity.
  - apathy 10y ago
    Ah, OK. I think we are on the same page. My mistake.

Jweb_Guru 10y ago

Bluntly, no. Laypeople are not the intended audience for scientific papers and usually lack the background to provide an informed review (hell, even most researchers in the field do). They do lack certain biases and incentives that researchers familiar with the topic usually have, but that doesn't compensate for what they don't know (for example, whether the claimed phenomenon has been tested many other times and not shown up; whether the claimed mechanism is bunk; whether the supposedly novel idea has already been published many other times). That being said, laypeople can (and should) absolutely call out research malpractice like what's described in the article.

wjnc 10y ago
I disagree pretty strongly. You've got worldwide universities churning out masters and pdh-level educated persons, who mostly go to the private sector for employment. Many of those would be perfectly able to perform peer review. And with this p-hacking crisis, mostly due to abject statistical knowledge, there are literally millions of technically able non-scientists able to review this nonsense. Any physicist, econometricist, statistician (and >100 more technical fields) could kindly take a shot at the social science methods in dispute here.
- Jweb_Guru 10y ago
  You're focusing on exactly one aspect of peer review, determining whether the statistical methods used were valid, which I agree can be done by pretty much anyone with a background in statistics (given that these days it's pretty easy to get in touch with the authors to ask for clarification). But peer review is about more than just determining whether the paper is internally coherent (or directly lifts text from other papers by the same author, which can be determined by a machine). I shouldn't have to say this, but a former physics Ph.D. is emphatically not qualified to perform peer review on papers on cell biology solely because physics and cell biology are both science. Papers don't exist in a vacuum, and by engaging in practices like selective citation (and others I mentioned in the previous post), it's possible to convince intelligent and reasonable people of a point that is completely wrong. In fact, it's so easy it even happens somewhat routinely within narrow subspecialties, where there may only be a handful of experts on a phenomenon worldwide. People who aren't up to date with the field already usually don't have time to spend weeks, months, or years trudging through the literature to gain enough background knowledge to make those kinds of decisions.