4 ms·

Peer review is not designed to detect fraud. It is meant to improve the quality of work, by giving a publication a fresh set of eyes who might identify gaps in

Peer review is not designed to detect fraud. It is meant to improve the quality of work, by giving a publication a fresh set of eyes who might identify gaps in the original research.

strangattractor 3y ago
Not having data is quite a significant gap. BTW: I am not criticizing the actual reviewer of the paper. Data availability should be a policy of the journal and required for publication. How can you improve quality without access to the data? In this instance requiring data would have likely prevented the paper from even reaching peer review and wasting a busy individuals time.
- 0cf8612b2e1e 3y ago
  That was one particular dataset being requested. There are near infinite ways to characterize a material, and even the best lab is only going to have the time, resources, and expertise to collect and present some of them. I am not in the field, so I cannot comment on the specific relevance of the request, but in my area, there are absolutely some mandatory data inclusions which should block a paper from being published. Presumably this was not on that list.
  - strangattractor 3y ago
    There are numerous ways to embargo data and make it available to reviewers before publication and to the public after publication. [1] This isn't a new idea. They simply did not do it. Nature is a "premier journal" known for publishing cutting edge research. Any data - in this case a bunch of numbers - used to support the conclusion should be made available. I am sure there are disciplines where the evidence would be difficult to provide directly - fossils, physical items etc. Maybe this PI did the experiment himself - thought he won a Noble - only to find he could not in fact reproduce the results after publication - oops. IMO Nature did not do its job. Just pointing that out. [1] https://plos.org/resource/how-to-store-and-manage-your-data/ https://plos.org/resource/how-to-store-and-manage-your-data/
- BeetleB 3y ago
  > Not having data is quite a significant gap. Requiring (raw) data is relatively new, and is only in a few fields. For most of the history of peer review data was not required. In fact, none of the PIs I worked in during grad school would ever agree to releasing the raw data. I'm pretty sure they're still not in most of physics. IMO, it's reasonable to require it, but understand that this is a relatively new requirement and not part of peer review culture.
  - worik 3y ago
    > Requiring (raw) data is relatively new, and is only in a few fields. That on its own is scandalous I was reading of the "reproducibility crisis", in computing, in the nineties Science still wants to be the warm friendly club, whilst playing competitively for high personal stakes. Cannot be both. Fraud, deliberately cynical fraud and hopeful naive fraud, has been rampant for decades The journals benefit directly and seem to be utterly complicit. The system is dreadfully broken Nobody should wonder why people do not trust science
    - imchillyb 3y ago
      Science is a method. Science isn't a religion, nor is it a paper, thesis, or journal. It's a method for understanding the world around us through rigorous application of testing and retesting. Science can be trusted. Measurements and the testing and retesting can be trusted. It's people that can't be trusted. We call people in the field of Science, Scientists. Perhaps we should stop labeling fallible and untrustworthy humans as such. Science is trustworthy. Humans are not.
      - ants_everywhere 3y ago
        > Science can be trusted. Measurements and the testing and retesting can be trusted. It's people that can't be trusted. This is an important point and I think people should make it more often. A human scientist who is committing fraud is misrepresenting facts about the world. And the way you would accurately describe those facts is with science. It's not that science is in the wrong here, it's that the human is lying about what the science says. You want to be able to say "objectively, this guy lied about his research." But if you want to say that, then you need to be able to accurately describe both the world and the research so you can compute the diff between them. So if you throw out the science with the human scientist, then you've thrown out any objective means by which you were going to say the human scientist did anything wrong in the first place.
pdonis 3y ago
> Peer review is not designed to detect fraud. Maybe it should be. "Peer reviewed" is taken as a sign of reliability in many contexts, including public policy. If it isn't actually such a sign, then either it needs to change so it is, or we need to stop taking it as a sign of something it isn't.
- 0cf8612b2e1e 3y ago
  I am not sure you appreciate the scope of distrusting submissions. Does this mean the reviewer has to clean-slate replicate everything? What if the author spent three years breeding a particular mouse model? Purified 8pg of an antibody extracted from flea tears? Spent six months of HPC time to simulate molecular dynamics? Some science is hard and incredibly difficult and expensive to do. Even just reevaluating a submitted dataset is no guarantee. Someone committing an elaborate fraud could have faked the raw datasets as well.
  - pdonis 3y ago
    > I am not sure you appreciate the scope of distrusting submissions. What you're saying is, the first option I gave, to fix the peer review process so it is a good indicator of the reliability of research, is not feasible. If that's the case, that just means we have to pick the second option: stop treating peer reviewed research as if it is reliable.
    - beowulfey 3y ago
      That's actually exactly correct. Peer review is supposed to characterize the research in terms of quality, not validate it. Otherwise, those peers who validate it would practically be co-authors. It's a system built upon honesty. A dishonest person is obviously going to take advantage of it. Really blatant things are caught in peer review, but sneaky fraud will definitely slip through.
      - pdonis 3y ago
        > Peer review is supposed to characterize the research in terms of quality Fraudulent research would obviously not be of the desired quality, wouldn't it? > It's a system built upon honesty. But if the purpose is to characterize the research in terms of quality, this can't be right. Since fraud is known to happen, and there are known to be incentives for fraud, any process that claims to judge the quality of research would have to have some way of checking for fraud.

Peer review is not designed to detect fraud. It is meant to improve the quality of work, by giving a publication a fresh set of eyes who might identify gaps in the original research.

strangattractor 3y ago

Not having data is quite a significant gap. BTW: I am not criticizing the actual reviewer of the paper. Data availability should be a policy of the journal and required for publication. How can you improve quality without access to the data? In this instance requiring data would have likely prevented the paper from even reaching peer review and wasting a busy individuals time.

0cf8612b2e1e 3y ago
That was one particular dataset being requested. There are near infinite ways to characterize a material, and even the best lab is only going to have the time, resources, and expertise to collect and present some of them. I am not in the field, so I cannot comment on the specific relevance of the request, but in my area, there are absolutely some mandatory data inclusions which should block a paper from being published. Presumably this was not on that list.
- strangattractor 3y ago
  There are numerous ways to embargo data and make it available to reviewers before publication and to the public after publication. [1] This isn't a new idea. They simply did not do it. Nature is a "premier journal" known for publishing cutting edge research. Any data - in this case a bunch of numbers - used to support the conclusion should be made available. I am sure there are disciplines where the evidence would be difficult to provide directly - fossils, physical items etc. Maybe this PI did the experiment himself - thought he won a Noble - only to find he could not in fact reproduce the results after publication - oops. IMO Nature did not do its job. Just pointing that out. [1] https://plos.org/resource/how-to-store-and-manage-your-data/ https://plos.org/resource/how-to-store-and-manage-your-data/
BeetleB 3y ago
> Not having data is quite a significant gap. Requiring (raw) data is relatively new, and is only in a few fields. For most of the history of peer review data was not required. In fact, none of the PIs I worked in during grad school would ever agree to releasing the raw data. I'm pretty sure they're still not in most of physics. IMO, it's reasonable to require it, but understand that this is a relatively new requirement and not part of peer review culture.
- worik 3y ago
  > Requiring (raw) data is relatively new, and is only in a few fields. That on its own is scandalous I was reading of the "reproducibility crisis", in computing, in the nineties Science still wants to be the warm friendly club, whilst playing competitively for high personal stakes. Cannot be both. Fraud, deliberately cynical fraud and hopeful naive fraud, has been rampant for decades The journals benefit directly and seem to be utterly complicit. The system is dreadfully broken Nobody should wonder why people do not trust science
  - imchillyb 3y ago
    Science is a method. Science isn't a religion, nor is it a paper, thesis, or journal. It's a method for understanding the world around us through rigorous application of testing and retesting. Science can be trusted. Measurements and the testing and retesting can be trusted. It's people that can't be trusted. We call people in the field of Science, Scientists. Perhaps we should stop labeling fallible and untrustworthy humans as such. Science is trustworthy. Humans are not.
    - ants_everywhere 3y ago
      > Science can be trusted. Measurements and the testing and retesting can be trusted. It's people that can't be trusted. This is an important point and I think people should make it more often. A human scientist who is committing fraud is misrepresenting facts about the world. And the way you would accurately describe those facts is with science. It's not that science is in the wrong here, it's that the human is lying about what the science says. You want to be able to say "objectively, this guy lied about his research." But if you want to say that, then you need to be able to accurately describe both the world and the research so you can compute the diff between them. So if you throw out the science with the human scientist, then you've thrown out any objective means by which you were going to say the human scientist did anything wrong in the first place.

pdonis 3y ago

> Peer review is not designed to detect fraud. Maybe it should be. "Peer reviewed" is taken as a sign of reliability in many contexts, including public policy. If it isn't actually such a sign, then either it needs to change so it is, or we need to stop taking it as a sign of something it isn't.

0cf8612b2e1e 3y ago
I am not sure you appreciate the scope of distrusting submissions. Does this mean the reviewer has to clean-slate replicate everything? What if the author spent three years breeding a particular mouse model? Purified 8pg of an antibody extracted from flea tears? Spent six months of HPC time to simulate molecular dynamics? Some science is hard and incredibly difficult and expensive to do. Even just reevaluating a submitted dataset is no guarantee. Someone committing an elaborate fraud could have faked the raw datasets as well.
- pdonis 3y ago
  > I am not sure you appreciate the scope of distrusting submissions. What you're saying is, the first option I gave, to fix the peer review process so it is a good indicator of the reliability of research, is not feasible. If that's the case, that just means we have to pick the second option: stop treating peer reviewed research as if it is reliable.
  - beowulfey 3y ago
    That's actually exactly correct. Peer review is supposed to characterize the research in terms of quality, not validate it. Otherwise, those peers who validate it would practically be co-authors. It's a system built upon honesty. A dishonest person is obviously going to take advantage of it. Really blatant things are caught in peer review, but sneaky fraud will definitely slip through.
    - pdonis 3y ago
      > Peer review is supposed to characterize the research in terms of quality Fraudulent research would obviously not be of the desired quality, wouldn't it? > It's a system built upon honesty. But if the purpose is to characterize the research in terms of quality, this can't be right. Since fraud is known to happen, and there are known to be incentives for fraud, any process that claims to judge the quality of research would have to have some way of checking for fraud.