3 ms·

Can anyone shed light on how a paper is published? I am guessing the process requires some higher authority to review the paper to check for legitimacy. If so,

Can anyone shed light on how a paper is published? I am guessing the process requires some higher authority to review the paper to check for legitimacy. If so, he committed fraud in 55 of his papers, not 1 or 2! So what irked me while I was reading is, "how was fraud missed so many times?" Someone else said that this is common in academia...if so, something HAS to be done to stop it. That's why I would like to know the process behind a paper being successfully published--it seems like a broken system if fraud like this occurs remotely often.

logjam 13y ago
Papers are generally peer-reviewed. However, this nitwit was just outright falsifying his data - peer-review or a "higher authority" review wouldn't necessarily have caught cooked data. Open data policies may help.
- eli 13y ago
  Even still it just would have made it more work to fake more convincing raw data. Based on the article I don't get the sense this would have dissuaded him.
shmageggy 13y ago
Typically papers are reviewed by anonymous peers (other researchers in the field) and the journal's editors. This is why the fraud review board chastised the field of Psychology as a whole. Note that it would have been nearly impossible to detect fraud in a single, isolated paper His data were fabricated in such a way as to seem believable, and it's not like researchers are expected to record video evidence of their experiments actually taking place or anything like that; they just report the results. He was ultimately revealed by those who were familiar with his work noticing patterns that appeared over many papers. I don't think there is any formal mechanism in place to review entire bodies of work as a whole.
- lmm 13y ago
  Missing raw data should surely be a big red flag; maybe not video but surely you're expected to include the individual response numbers so that other people can check your analysis, and it sounds like he claimed to not even have those.
  - jorleif 13y ago
    Usually these kinds of data are not part of the publication or up for review. I don't know why exactly, but I suppose it is a combination of: 1) paper publishing tradition, where distributing this material is too much work. 2) too much work for reviewers to comb through raw data, the real solution would be to require sharing of the raw data, so readers rather than the reviewers would find mistakes, but this is often difficult because of privacy issues. E.g. videos of the Utrecht train station study would make fraud more difficult, but publishing the videos would probably require permission from all participants 3) Trust in the authors' integrity. Most scientists would not make up data, but they might make logical errors in their reasoning, or use bad procedures. Peer review can find these errors, but if the authors lie about their procedures, it is very hard to check. Peer review is about internal consistency.
denzil_correa 13y ago
> it seems like a broken system if fraud like this occurs remotely often That's the problem. It indeed is broken and not just in terms of fraud; in more ways than one but people who "game the system" call the shots and wouldn't like to fix it.
adharmad 13y ago
Psychology being an inexact science, it is difficult for someone else to repeat the same experiment and check the validity of the results (such as in the physical sciences). Even if someone does get different numbers, the validity of the repeat experiment itself can be questioned, or it can be put down to any number of variables. eg - the human subjects themselves will not the same.
- jorleif 13y ago
  The much bigger problem is that nobody tries to repeat findings, because there is no incentive to do so. There is an asymmetry between positive and negative findings. Say I make a study that shows that X causes Y. Then someone else tries to replicate that. They find no statistically significant evidence that X causes Y. Now, in medicine this may be a result in itself, if X causes Y is a well established result. But, in many other fields, such as computer science, a negative finding is usually just because the researcher made a mistake (bug in code etc.), so you have to work really hard to prove that there is no relation between X and Y, and it is hard to publish these kinds of results unless you can somehow rigorously prove the opposite result.

logjam 13y ago

Papers are generally peer-reviewed. However, this nitwit was just outright falsifying his data - peer-review or a "higher authority" review wouldn't necessarily have caught cooked data. Open data policies may help.

eli 13y ago
Even still it just would have made it more work to fake more convincing raw data. Based on the article I don't get the sense this would have dissuaded him.

shmageggy 13y ago

Typically papers are reviewed by anonymous peers (other researchers in the field) and the journal's editors. This is why the fraud review board chastised the field of Psychology as a whole. Note that it would have been nearly impossible to detect fraud in a single, isolated paper His data were fabricated in such a way as to seem believable, and it's not like researchers are expected to record video evidence of their experiments actually taking place or anything like that; they just report the results. He was ultimately revealed by those who were familiar with his work noticing patterns that appeared over many papers. I don't think there is any formal mechanism in place to review entire bodies of work as a whole.

lmm 13y ago
Missing raw data should surely be a big red flag; maybe not video but surely you're expected to include the individual response numbers so that other people can check your analysis, and it sounds like he claimed to not even have those.
- jorleif 13y ago
  Usually these kinds of data are not part of the publication or up for review. I don't know why exactly, but I suppose it is a combination of: 1) paper publishing tradition, where distributing this material is too much work. 2) too much work for reviewers to comb through raw data, the real solution would be to require sharing of the raw data, so readers rather than the reviewers would find mistakes, but this is often difficult because of privacy issues. E.g. videos of the Utrecht train station study would make fraud more difficult, but publishing the videos would probably require permission from all participants 3) Trust in the authors' integrity. Most scientists would not make up data, but they might make logical errors in their reasoning, or use bad procedures. Peer review can find these errors, but if the authors lie about their procedures, it is very hard to check. Peer review is about internal consistency.

denzil_correa 13y ago

> it seems like a broken system if fraud like this occurs remotely often That's the problem. It indeed is broken and not just in terms of fraud; in more ways than one but people who "game the system" call the shots and wouldn't like to fix it.

adharmad 13y ago

Psychology being an inexact science, it is difficult for someone else to repeat the same experiment and check the validity of the results (such as in the physical sciences). Even if someone does get different numbers, the validity of the repeat experiment itself can be questioned, or it can be put down to any number of variables. eg - the human subjects themselves will not the same.

jorleif 13y ago
The much bigger problem is that nobody tries to repeat findings, because there is no incentive to do so. There is an asymmetry between positive and negative findings. Say I make a study that shows that X causes Y. Then someone else tries to replicate that. They find no statistically significant evidence that X causes Y. Now, in medicine this may be a result in itself, if X causes Y is a well established result. But, in many other fields, such as computer science, a negative finding is usually just because the researcher made a mistake (bug in code etc.), so you have to work really hard to prove that there is no relation between X and Y, and it is hard to publish these kinds of results unless you can somehow rigorously prove the opposite result.