3 ms·

I've been more disturbed by the response to this than the papers themselves. Saying,well of course we were duped, you really expected the peer review system to

I've been more disturbed by the response to this than the papers themselves. Saying,well of course we were duped, you really expected the peer review system to not to utterly fail in the presence of a few bad actors? does not inspire my faith in the system.

eindiran 8y ago
That's what stuck out to me as well. This quote: "I am so utterly unimpressed," wrote Jacob T. Levy, a political theorist at McGill University, "by the fact that an enterprise that relies on a widespread presumption of not-fraud can be fooled some of the time by three people with Ph.D.s who spend 10 months deliberately trying to defraud it." Isn't the ability to separate out the quality work from the garbage the point of the peer review system? If the system was working as intended, the reviewers even with the assumption that they aren't being defrauded should have been able to sort the papers into the garbage pile.
- davemp 8y ago
  > an enterprise that relies on a widespread presumption of not-fraud This is section really gets me. How in the world is science supposed to rely on “not-fraud”? Sure if you fake your data you might dupe a diligent reviewer, but this is why results are supposed to be reproduced. Anything that relies on “not-fraud” seems inherently unscientific and straight up naive.
  - BurritoAlPastor 8y ago
    Social sciences have pretty deep-seated logistical obstacles to reproducibility. If you want to reproduce an archaeology paper, well, how? You can’t exactly go dig the same site back up. If you go out as a cultural anthropologist to hang out with some indigenous tribe somewhere, and your experiences aren’t the same as the last guy, does that mean anything? How much is because you’re different people? How much is because the tribe has already gotten used to having a researcher around? This is part of why shit like the Piltdown Man happens.
  - mooman219 8y ago
    Exactly this. If my banking API relies on user's not trying to commit fraud instead of proper validation, then I have a very insecure API. Making a statement that your system works as long as no one tries to break it sounds almost satirical.
    - amw 8y ago
      The result of successful banking API fraud is "I get a lot of money". The result of successful science publication fraud is, "I get some papers published, and as long as I don't get too famous and people don't look too hard at replicating my results later, maybe I get to include them in a bibliography or resume somewhere".
  - TheCoelacanth 8y ago
    Reproduction of results happens after publication, though. Simply getting published does not demonstrate that their ideas were accepted as truthful, you should look at things like whether their work was cited by other work.
  - amw 8y ago
    It's supposed to rely on not-fraud because its incentive structure is not supposed to reward gaming the system (the way that "you get money" incentivizes fraud in a market system). A certain amount of good faith is presumed because the people who are eventually found retracted en mass will have a rough time getting work in the future, and because they won't be any closer to getting work published that will make them well-known scientists in the future.
  - tptacek 8y ago
    Sussing out invalid research is the purpose of replication. A replication is itself a research project. It's not something that peer reviewers do.
    - davemp 8y ago
      Yes. My comment implies that you can get invalid research published. At least it was intended to. The problem lies in areas where replication and formalization is difficult. There is certainly knowledge to be gained in these areas. Though calling the fields sciences, when the scientific method cannot be effectively applied, seems dubious. The solution is not obvious, but it is clear that steps should be taken to improve rigor in the less formal (straight forward) bodies of scholarship.
      - foldr 8y ago
        It isn't possible to attempt replication of every scientific paper that's published. A lot of people on this thread seem to have difficulty accepting this fact, but the time and money to do that simply isn't available. If a large percentage of scientists were to engage in deliberate fraud, there wouldn't be any feasible way to regulate that problem away. All available resources would end up being spent on scientists investigating each other for fraud, and no actual science would get done. Bear in mind that in many cases, establishing that an experiment is not fraudulent could require more time and effort than the original experiment! As with many other human institutions, we have no choice but to rely on most people being honest most of the time.
- Trombone12 8y ago
  You better pay reviewers anything at all if you want them to produce few type 2 errors.
- TheOtherHobbes 8y ago
  The point of the peer review system is to advance careers, not to produce high quality scholarship. There are two separate issues here. One is that peer review fails in disciplines that are largely bogus and based on mimicry and posturing. IMO this absolutely applies to a lot of academic social sciences - but it also applies equally, allowing for changes in tone, convention and social register, to their ideological equivalents on the right. The other is that we really have a crisis of authority and competence in general. Between fake news, fake reviews, more and less professional trolls and manipulators, empty academic tribal signalling, wilfully dishonest fraud in professional science, the entire PR and thinktank industry of spin, manipulation, and lies for influence, and the slightly milder but still toxic ad industry, there's far too little useful signal now among the tsunami of noise that we have to sift through every day.
- eridius 8y ago
  Separating out the quality work from the garbage is different than setting out the work done in good faith from the work done in bad faith by people who are otherwise capable of producing good work.
- ModernMech 8y ago
  > Isn't the ability to separate out the quality work from the garbage the point of the peer review system The peer review system is just one step in the scientific process. After passing the review it gets published and more scientists see it. They can then tear it down or build on top of it. If it's actually bad science, it probably won't survive long.
- UncleMeat 8y ago
  Not at all. Having reviewed papers, my goal is to make the underlying experiment as clear as possible for readers when the paper is eventually published. I care about the methods being clear and the analysis being sound. I am absolutely not looking for "I think the authors are lying" warnings because this way lies madness. If we expect reviewers to reject papers they find weird or suspicious then a lot of excellent scholarship will be rejected because established researchers are stuck in their belief systems. It also leads to serious problems when an author has obtained some hard-to-get data set and this is now a downside when it comes to review. If the results are garbage then that will come out later when people work with the ideas presented in the paper.
  - adrianratnapala 8y ago
    That's because reviewers of experimental papers are not expected to reproduce experiments. In theoretical physics or mathematics a reviewer absolutely should follow the proofs and check they are sound. The humanities (as opposed to the social sciences) are theory fields, they don't have experiment but they advance knowledge by good arguments that require real scholarship to support even if they don't have crispness of proofs. Someone reviewing it should go "Yep, that paper make a real claim and argues for it well enough that convincing counter-argument would need to be a significant work of scholarship in itself". If a field of study produces a conversation made up of such arguments and counter-arguments, then it stands some chance of being growing repository of truth. Whereas if the field can't distinguish between it's own chatter and a hoax guided by mood affiliation, then the field is likely to be just chatter.
    - TheCoelacanth 8y ago
      If the field doesn't rely on experimentation, why did their papers include fake experimental data (e.g. one of the papers contained a bunch of fake data about their observations of dog genitalia)?
tptacek 8y ago
What did you believe peer review was before you learned about this? You couldn't have actually done peer review before, because literally everyone who participates in that system in academia understands that they're at an immense disadvantage against a bad-faith submission. For every accepted paper, there are usually many more rejects, and all of them have to get read, and all in the reviewer's spare time.

eindiran 8y ago

That's what stuck out to me as well. This quote: "I am so utterly unimpressed," wrote Jacob T. Levy, a political theorist at McGill University, "by the fact that an enterprise that relies on a widespread presumption of not-fraud can be fooled some of the time by three people with Ph.D.s who spend 10 months deliberately trying to defraud it." Isn't the ability to separate out the quality work from the garbage the point of the peer review system? If the system was working as intended, the reviewers even with the assumption that they aren't being defrauded should have been able to sort the papers into the garbage pile.

davemp 8y ago
> an enterprise that relies on a widespread presumption of not-fraud This is section really gets me. How in the world is science supposed to rely on “not-fraud”? Sure if you fake your data you might dupe a diligent reviewer, but this is why results are supposed to be reproduced. Anything that relies on “not-fraud” seems inherently unscientific and straight up naive.
- BurritoAlPastor 8y ago
  Social sciences have pretty deep-seated logistical obstacles to reproducibility. If you want to reproduce an archaeology paper, well, how? You can’t exactly go dig the same site back up. If you go out as a cultural anthropologist to hang out with some indigenous tribe somewhere, and your experiences aren’t the same as the last guy, does that mean anything? How much is because you’re different people? How much is because the tribe has already gotten used to having a researcher around? This is part of why shit like the Piltdown Man happens.
- mooman219 8y ago
  Exactly this. If my banking API relies on user's not trying to commit fraud instead of proper validation, then I have a very insecure API. Making a statement that your system works as long as no one tries to break it sounds almost satirical.
  - amw 8y ago
    The result of successful banking API fraud is "I get a lot of money". The result of successful science publication fraud is, "I get some papers published, and as long as I don't get too famous and people don't look too hard at replicating my results later, maybe I get to include them in a bibliography or resume somewhere".
- TheCoelacanth 8y ago
  Reproduction of results happens after publication, though. Simply getting published does not demonstrate that their ideas were accepted as truthful, you should look at things like whether their work was cited by other work.
- amw 8y ago
  It's supposed to rely on not-fraud because its incentive structure is not supposed to reward gaming the system (the way that "you get money" incentivizes fraud in a market system). A certain amount of good faith is presumed because the people who are eventually found retracted en mass will have a rough time getting work in the future, and because they won't be any closer to getting work published that will make them well-known scientists in the future.
- tptacek 8y ago
  Sussing out invalid research is the purpose of replication. A replication is itself a research project. It's not something that peer reviewers do.
  - davemp 8y ago
    Yes. My comment implies that you can get invalid research published. At least it was intended to. The problem lies in areas where replication and formalization is difficult. There is certainly knowledge to be gained in these areas. Though calling the fields sciences, when the scientific method cannot be effectively applied, seems dubious. The solution is not obvious, but it is clear that steps should be taken to improve rigor in the less formal (straight forward) bodies of scholarship.
    - foldr 8y ago
      It isn't possible to attempt replication of every scientific paper that's published. A lot of people on this thread seem to have difficulty accepting this fact, but the time and money to do that simply isn't available. If a large percentage of scientists were to engage in deliberate fraud, there wouldn't be any feasible way to regulate that problem away. All available resources would end up being spent on scientists investigating each other for fraud, and no actual science would get done. Bear in mind that in many cases, establishing that an experiment is not fraudulent could require more time and effort than the original experiment! As with many other human institutions, we have no choice but to rely on most people being honest most of the time.
Trombone12 8y ago
You better pay reviewers anything at all if you want them to produce few type 2 errors.
TheOtherHobbes 8y ago
The point of the peer review system is to advance careers, not to produce high quality scholarship. There are two separate issues here. One is that peer review fails in disciplines that are largely bogus and based on mimicry and posturing. IMO this absolutely applies to a lot of academic social sciences - but it also applies equally, allowing for changes in tone, convention and social register, to their ideological equivalents on the right. The other is that we really have a crisis of authority and competence in general. Between fake news, fake reviews, more and less professional trolls and manipulators, empty academic tribal signalling, wilfully dishonest fraud in professional science, the entire PR and thinktank industry of spin, manipulation, and lies for influence, and the slightly milder but still toxic ad industry, there's far too little useful signal now among the tsunami of noise that we have to sift through every day.
eridius 8y ago
Separating out the quality work from the garbage is different than setting out the work done in good faith from the work done in bad faith by people who are otherwise capable of producing good work.
ModernMech 8y ago
> Isn't the ability to separate out the quality work from the garbage the point of the peer review system The peer review system is just one step in the scientific process. After passing the review it gets published and more scientists see it. They can then tear it down or build on top of it. If it's actually bad science, it probably won't survive long.
UncleMeat 8y ago
Not at all. Having reviewed papers, my goal is to make the underlying experiment as clear as possible for readers when the paper is eventually published. I care about the methods being clear and the analysis being sound. I am absolutely not looking for "I think the authors are lying" warnings because this way lies madness. If we expect reviewers to reject papers they find weird or suspicious then a lot of excellent scholarship will be rejected because established researchers are stuck in their belief systems. It also leads to serious problems when an author has obtained some hard-to-get data set and this is now a downside when it comes to review. If the results are garbage then that will come out later when people work with the ideas presented in the paper.
- adrianratnapala 8y ago
  That's because reviewers of experimental papers are not expected to reproduce experiments. In theoretical physics or mathematics a reviewer absolutely should follow the proofs and check they are sound. The humanities (as opposed to the social sciences) are theory fields, they don't have experiment but they advance knowledge by good arguments that require real scholarship to support even if they don't have crispness of proofs. Someone reviewing it should go "Yep, that paper make a real claim and argues for it well enough that convincing counter-argument would need to be a significant work of scholarship in itself". If a field of study produces a conversation made up of such arguments and counter-arguments, then it stands some chance of being growing repository of truth. Whereas if the field can't distinguish between it's own chatter and a hoax guided by mood affiliation, then the field is likely to be just chatter.
  - TheCoelacanth 8y ago
    If the field doesn't rely on experimentation, why did their papers include fake experimental data (e.g. one of the papers contained a bunch of fake data about their observations of dog genitalia)?

tptacek 8y ago

What did you believe peer review was before you learned about this? You couldn't have actually done peer review before, because literally everyone who participates in that system in academia understands that they're at an immense disadvantage against a bad-faith submission. For every accepted paper, there are usually many more rejects, and all of them have to get read, and all in the reviewer's spare time.