4 ms·

> If you don't what someone did exactly and what they used then replicating it is going to be very difficult. Why do journals accept these papers? If you can j

by drb91 8y ago

> If you don't what someone did exactly and what they used then replicating it is going to be very difficult. Why do journals accept these papers? If you can just make shit up, it kind of makes the whole value of a paid journal moot. What value do they add if not reviewing the damn paper?

freetime2 8y ago
I don't write research papers, so I don't really know what I'm talking about here, but I assume it's not as easy to just "make shit up" as you are implying here. There's a peer review process, there are lots of people working in the lab that make it difficult to keep a large conspiracy secret, there are serious penalties if you are caught fabricating results, etc.
- drb91 8y ago
  There is a difference between wholesale making things up and conveniently leaving out variables that alter analysis of your data. I do agree overall that this is not trivial and there are tremendous disincentives; that is evidentally not a good tactic to reproducing experiments in the general case.
- stonemetal 8y ago
  What do the reviewers do? They don't try to reproduce it. So yeah make shit up, as long as it is plausible no one will notice. For example Yoshihiro Sato was recently (2016-2017 time frame) caught fabricating results, turns out he had been doing it for 30 years. He had more than 20 papers retracted. Also note he wasn't caught by a fellow scientist in the field he was caught by a statistician bulk processing papers looking for anomalies. So yeah big penalties if you are caught, but the odds of getting caught are so low you have to be stunningly stupid to get caught.
  - southern_cross 8y ago
    Sadly, once the statistical methods being used here and the types of anomalies being found become well-known, it's probably not too hard for fakers to adjust their "results" so that such anomalies are far more difficult (if not impossible) to detect. For example, if an analysis would show that your experimental data is maybe a bit too clean, in a way that wouldn't normally turn up in the real world and would therefore cast doubt on its validity, then you just need to make sure that you dirty it up a little before publication.
- jpeloquin 8y ago
  Peer review is pretty ineffective at catching errors. In one study of peer review, reviewers caught ~ 30% of deliberately inserted errors that were specifically chosen to be detectable in review [0]. Fraud is noticed by the broader research community (assuming it's noticed at all) when it continues for years over multiple manuscripts. You make a good point about colleagues in the same lab noticing unethical behavior. A shared research culture that values integrity and transparency is a good defense against malpractice and fraud. Building up and maintaining that culture—and the methodological skills to back it up—is a continual challenge. [0] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2586872/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2586872/
karthikb 8y ago
What is repeatability? 10% of the time? 50% of the time? 99% of the time? 99.9999% of the time? Even medications made in high volume processes have rejected units (and sometimes, batches). Process drifted, machine failed, raw ingredients had some issue. A scientific paper at the very early stage is meant to show that something is possible if the conditions are just right. It may be actually outside the capability of the lab to document (or measure, or even know) every variable that makes up these conditions. One of the reasons there is such a chasm between lab and practice is that as people dive in for deeper review and productization, they find the process is too finicky, the applicability too narrow, etc. But the scientific paper is the first step in this process of discovery. That said, it's critical that authors document whatever they do know or have control over. Electronic publishing, tools like Github, etc, all make the barrier for disclosure much lower.
- drb91 8y ago
  Presumably it would be context sensitive like most variables. Just because something is hard does not mean it is not worth doing. Confirmation is inherently impossible, but we should be able to weed out bad papers with higher accuracy.
  - karthikb 8y ago
    Absolutely agreed
- stonemetal 8y ago
  Given that P <= 0.05 is the usual standard, I would hazard that if it can't be reproduced at least 95% of the time then you miscalculated P.
  - kgwgk 8y ago
    This is no what a p-value means.
    - stonemetal 8y ago
deleted 8y ago
[deleted]
icelancer 8y ago
Because the journals in question are not open access / open data where data submission is required. To answer your next question, yes, that entirely goes against the whole point of science, which is that it's supposed to be reproducible.

by drb91 8y ago

freetime2 8y ago

I don't write research papers, so I don't really know what I'm talking about here, but I assume it's not as easy to just "make shit up" as you are implying here. There's a peer review process, there are lots of people working in the lab that make it difficult to keep a large conspiracy secret, there are serious penalties if you are caught fabricating results, etc.

drb91 8y ago
There is a difference between wholesale making things up and conveniently leaving out variables that alter analysis of your data. I do agree overall that this is not trivial and there are tremendous disincentives; that is evidentally not a good tactic to reproducing experiments in the general case.
stonemetal 8y ago
What do the reviewers do? They don't try to reproduce it. So yeah make shit up, as long as it is plausible no one will notice. For example Yoshihiro Sato was recently (2016-2017 time frame) caught fabricating results, turns out he had been doing it for 30 years. He had more than 20 papers retracted. Also note he wasn't caught by a fellow scientist in the field he was caught by a statistician bulk processing papers looking for anomalies. So yeah big penalties if you are caught, but the odds of getting caught are so low you have to be stunningly stupid to get caught.
- southern_cross 8y ago
  Sadly, once the statistical methods being used here and the types of anomalies being found become well-known, it's probably not too hard for fakers to adjust their "results" so that such anomalies are far more difficult (if not impossible) to detect. For example, if an analysis would show that your experimental data is maybe a bit too clean, in a way that wouldn't normally turn up in the real world and would therefore cast doubt on its validity, then you just need to make sure that you dirty it up a little before publication.
jpeloquin 8y ago
Peer review is pretty ineffective at catching errors. In one study of peer review, reviewers caught ~ 30% of deliberately inserted errors that were specifically chosen to be detectable in review [0]. Fraud is noticed by the broader research community (assuming it's noticed at all) when it continues for years over multiple manuscripts. You make a good point about colleagues in the same lab noticing unethical behavior. A shared research culture that values integrity and transparency is a good defense against malpractice and fraud. Building up and maintaining that culture—and the methodological skills to back it up—is a continual challenge. [0] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2586872/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2586872/

karthikb 8y ago

What is repeatability? 10% of the time? 50% of the time? 99% of the time? 99.9999% of the time? Even medications made in high volume processes have rejected units (and sometimes, batches). Process drifted, machine failed, raw ingredients had some issue. A scientific paper at the very early stage is meant to show that something is possible if the conditions are just right. It may be actually outside the capability of the lab to document (or measure, or even know) every variable that makes up these conditions. One of the reasons there is such a chasm between lab and practice is that as people dive in for deeper review and productization, they find the process is too finicky, the applicability too narrow, etc. But the scientific paper is the first step in this process of discovery. That said, it's critical that authors document whatever they do know or have control over. Electronic publishing, tools like Github, etc, all make the barrier for disclosure much lower.

drb91 8y ago
Presumably it would be context sensitive like most variables. Just because something is hard does not mean it is not worth doing. Confirmation is inherently impossible, but we should be able to weed out bad papers with higher accuracy.
- karthikb 8y ago
  Absolutely agreed
stonemetal 8y ago
Given that P <= 0.05 is the usual standard, I would hazard that if it can't be reproduced at least 95% of the time then you miscalculated P.
- kgwgk 8y ago
  This is no what a p-value means.
  - stonemetal 8y ago

deleted 8y ago

[deleted]

icelancer 8y ago

Because the journals in question are not open access / open data where data submission is required. To answer your next question, yes, that entirely goes against the whole point of science, which is that it's supposed to be reproducible.