4 ms·

I don't write research papers, so I don't really know what I'm talking about here, but I assume it's not as easy to just "make shit up" as you are implying here

I don't write research papers, so I don't really know what I'm talking about here, but I assume it's not as easy to just "make shit up" as you are implying here. There's a peer review process, there are lots of people working in the lab that make it difficult to keep a large conspiracy secret, there are serious penalties if you are caught fabricating results, etc.

drb91 8y ago
There is a difference between wholesale making things up and conveniently leaving out variables that alter analysis of your data. I do agree overall that this is not trivial and there are tremendous disincentives; that is evidentally not a good tactic to reproducing experiments in the general case.
stonemetal 8y ago
What do the reviewers do? They don't try to reproduce it. So yeah make shit up, as long as it is plausible no one will notice. For example Yoshihiro Sato was recently (2016-2017 time frame) caught fabricating results, turns out he had been doing it for 30 years. He had more than 20 papers retracted. Also note he wasn't caught by a fellow scientist in the field he was caught by a statistician bulk processing papers looking for anomalies. So yeah big penalties if you are caught, but the odds of getting caught are so low you have to be stunningly stupid to get caught.
- southern_cross 8y ago
  Sadly, once the statistical methods being used here and the types of anomalies being found become well-known, it's probably not too hard for fakers to adjust their "results" so that such anomalies are far more difficult (if not impossible) to detect. For example, if an analysis would show that your experimental data is maybe a bit too clean, in a way that wouldn't normally turn up in the real world and would therefore cast doubt on its validity, then you just need to make sure that you dirty it up a little before publication.
jpeloquin 8y ago
Peer review is pretty ineffective at catching errors. In one study of peer review, reviewers caught ~ 30% of deliberately inserted errors that were specifically chosen to be detectable in review [0]. Fraud is noticed by the broader research community (assuming it's noticed at all) when it continues for years over multiple manuscripts. You make a good point about colleagues in the same lab noticing unethical behavior. A shared research culture that values integrity and transparency is a good defense against malpractice and fraud. Building up and maintaining that culture—and the methodological skills to back it up—is a continual challenge. [0] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2586872/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2586872/

drb91 8y ago

There is a difference between wholesale making things up and conveniently leaving out variables that alter analysis of your data. I do agree overall that this is not trivial and there are tremendous disincentives; that is evidentally not a good tactic to reproducing experiments in the general case.

stonemetal 8y ago

What do the reviewers do? They don't try to reproduce it. So yeah make shit up, as long as it is plausible no one will notice. For example Yoshihiro Sato was recently (2016-2017 time frame) caught fabricating results, turns out he had been doing it for 30 years. He had more than 20 papers retracted. Also note he wasn't caught by a fellow scientist in the field he was caught by a statistician bulk processing papers looking for anomalies. So yeah big penalties if you are caught, but the odds of getting caught are so low you have to be stunningly stupid to get caught.

southern_cross 8y ago
Sadly, once the statistical methods being used here and the types of anomalies being found become well-known, it's probably not too hard for fakers to adjust their "results" so that such anomalies are far more difficult (if not impossible) to detect. For example, if an analysis would show that your experimental data is maybe a bit too clean, in a way that wouldn't normally turn up in the real world and would therefore cast doubt on its validity, then you just need to make sure that you dirty it up a little before publication.

jpeloquin 8y ago

Peer review is pretty ineffective at catching errors. In one study of peer review, reviewers caught ~ 30% of deliberately inserted errors that were specifically chosen to be detectable in review [0]. Fraud is noticed by the broader research community (assuming it's noticed at all) when it continues for years over multiple manuscripts. You make a good point about colleagues in the same lab noticing unethical behavior. A shared research culture that values integrity and transparency is a good defense against malpractice and fraud. Building up and maintaining that culture—and the methodological skills to back it up—is a continual challenge. [0] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2586872/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2586872/