9 ms·

Use of 'language of deceit' betrays scientific fraud

19 points by Libertatea 12y ago

lutusp 12y ago
A quote: "Diederik Stapel, the infamous "lying Dutchman" who in 2011 admitted to inventing the data in dozens of psychology research papers, unwittingly signalled his deceit through the language he used. As well as inflating the certainty surrounding his results, Stapel included more science-related terms to describe his methods when writing up his fraudulent "findings" than when describing genuine results. "Researchers who have analysed Stapel's papers say they can separate his genuine research from the fictional with about 70 per cent accuracy. Now they are studying a larger sample of papers from many different scientific fraudsters, to see if the detection method works more generally." Just think -- if psychology were a science, researchers could use scientific tools to detect fraud -- tests that were not conducted, lab measurements that were fudged, images manipulated to create a false impression, as happens in real scientific fields. In extreme cases, we could detect fraud by simply repeating the experiments, using the clear protocols published along with the results. But in psychology, investigators are completely dependent on what people say, not what they do. If Stapel were in a real scientific field, he would never have gotten away with his many verbal frauds, and fraud detection would be child's play. Another quote: "Stapel, who worked at Tilburg University in the Netherlands, used more "amplifiers" – words like "profoundly" and "extreme" – in his fraudulent papers, and fewer "diminishers" – like "merely" and "somewhat"." Imagine a real scientist trying to defraud his university and granting agencies by using words like "profoundly" and "extreme" about his lab results. But in science, such descriptions would make no difference -- the lab results, the evidence, would decide the issue. It's important to understand that in psychology, because of its status as a pseudoscience, how you describe your work is more important than the outcome of experiments, and one can cheat with words. In a real science, Diederik Stapel would have been exposed within days of entering a laboratory.
- deleted 12y ago
  [deleted]
  - lutusp 12y ago
    > You ignore the use of the detection method in other areas - areas that you would call science. The linked article suggests that psychologists may try to produce results by analyzing how actual scientists talk about their science, blithely ignoring the fact that examining the science itself would be more effective. Also, an advocate of what you suggest would be hard-pressed to produce tangible evidence for results in mainstream science, because there aren't any. Frauds in real scientific fields are exposed, not by comparing their words to their work, but by comparing the evidence to reality. > Do you realise just how kook-like your behaviour is when ever the subject approaches psychiatry? 1. Are you aware that your arguments have no content, nothing that might be mistaken for science or good-faith debate, as a result of which you're making psychology look even more ridiculous that it already does? 2. Are you aware that my views on this topic now represent the mainstream? http://www.huffingtonpost.com/dr-harold-koplewicz/dsm-mental-health-research_b_3247960.html http://www.huffingtonpost.com/dr-harold-koplewicz/dsm-mental... Ten years ago, I started a campaign against the use of Asperger Syndrome as a diagnosis, arguing that it was a particularly absurd diagnosis, able to reliably "diagnose" bright kids as mentally ill. After years of hearing from people like you who know nothing about the topic under discussion, Asperger Syndrome was abandoned for the reasons I gave. In this new effort, the DSM is being challenged for its lack of scientific rigor, and not just by me, but by people like the current chairman of the NIMH. Conclusion? You need a reality check. If psychologists were to read your replies to my posts, they would say, "Please -- don't be on our side."
    - PhasmaFelis 12y ago
      > Ten years ago, I started a campaign against the use of Asperger Syndrome as a diagnosis, arguing that it was a particularly absurd diagnosis, able to reliably "diagnose" bright kids as mentally ill. After years of hearing from people like you who know nothing about the topic under discussion, Asperger Syndrome was abandoned for the reasons I gave. Asperger's wasn't "abandoned," it was folded into the broader spectrum of autism--and not by the field in general but only by the DSM, which you seem to distrust specifically, so I'm not quite sure where you're going with this claim. My Asperger's/autism diagnosis has made a huge difference to my quality of life, by helping me understand, deal with, and find support for my particular social, sensory, and mental-processing issues. The same is true for thousands of others, and we're kind of tired of you psychology deniers telling us that we don't exist.
- DanBC 12y ago
  > In a real science, Diederik Stapel would have been exposed within days of entering a laboratory. You ignore the fact that scientific fraud exists in other sciences; and that this article talks about extending the use of this tool to different sciences.
  - lutusp 12y ago
    > You ignore the fact that scientific fraud exists in other sciences It's not the topic under discussion, but scientific fraud certainly exists among scientists, with one important difference -- real scientists are exposed by comparing their work, their evidence, to reality. Stapel was exposed by comparing his words to his work -- work often not even conducted. There was never any serious prospect of comparing Stapel's work to reality, because his work didn't try to address reality -- that's science's purview. In science, reliant on physical evidence, how you describe the evidence shouldn't matter, the evidence should speak for itself. Reliable evidence should bring different, similarly equipped observers to the same conclusion. In psychology, the reason there's no consensus on the meaning of evidence is because the evidence is extremely poor, freeing different psychologists to draw different conclusions from the same evidence. > this article talks about extending the use of this tool to different sciences. Yes, that's true, they do talk about it. But until they actually try to apply it, it's not a legitimate topic of discussion. You haven't asked an obvious question -- where are the falsifiable psychological theories, theories that can be proven false in practical tests, but that resist falsification? Where are the time-tested theories that would make psychology one field, like physics or biology, and that would justify clinical practice?
shawn-furyan 12y ago
A 30% error rate is far, far too large for this to be useful as a first line filter. Even error rates of less than 1/3 of that (10%) lead to combinatorial effects that make for very unreliable tests. At best, this would be a corroborating test, and not a very strong one at that.
- rflrob 12y ago
  What I, if I were a reviewer on this paper, would have liked to have seen is a ROC curve—basically, if you tweak the cut offs, can you get fewer false positives at the expense of more false negatives. Assuming fraud in science is rare (I think even the gloom-and-doom-iest reports put it under 20% for any kind of research misconduct, not just fraud), even a moderately low false positive rate will flag a lot more papers than a perfect true positive rate.
- a3n 12y ago
  Think of it as early days.
rflrob 12y ago
It seems to me that every researcher will have a different level of 'amplifiers' in their writing, so this will be a really hard classifier to train generally.

19 points by Libertatea 12y ago

A quote: "Diederik Stapel, the infamous "lying Dutchman" who in 2011 admitted to inventing the data in dozens of psychology research papers, unwittingly signalled his deceit through the language he used. As well as inflating the certainty surrounding his results, Stapel included more science-related terms to describe his methods when writing up his fraudulent "findings" than when describing genuine results. "Researchers who have analysed Stapel's papers say they can separate his genuine research from the fictional with about 70 per cent accuracy. Now they are studying a larger sample of papers from many different scientific fraudsters, to see if the detection method works more generally." Just think -- if psychology were a science, researchers could use scientific tools to detect fraud -- tests that were not conducted, lab measurements that were fudged, images manipulated to create a false impression, as happens in real scientific fields. In extreme cases, we could detect fraud by simply repeating the experiments, using the clear protocols published along with the results. But in psychology, investigators are completely dependent on what people say, not what they do. If Stapel were in a real scientific field, he would never have gotten away with his many verbal frauds, and fraud detection would be child's play. Another quote: "Stapel, who worked at Tilburg University in the Netherlands, used more "amplifiers" – words like "profoundly" and "extreme" – in his fraudulent papers, and fewer "diminishers" – like "merely" and "somewhat"." Imagine a real scientist trying to defraud his university and granting agencies by using words like "profoundly" and "extreme" about his lab results. But in science, such descriptions would make no difference -- the lab results, the evidence, would decide the issue. It's important to understand that in psychology, because of its status as a pseudoscience, how you describe your work is more important than the outcome of experiments, and one can cheat with words. In a real science, Diederik Stapel would have been exposed within days of entering a laboratory.

deleted 12y ago
[deleted]
- lutusp 12y ago
  > You ignore the use of the detection method in other areas - areas that you would call science. The linked article suggests that psychologists may try to produce results by analyzing how actual scientists talk about their science, blithely ignoring the fact that examining the science itself would be more effective. Also, an advocate of what you suggest would be hard-pressed to produce tangible evidence for results in mainstream science, because there aren't any. Frauds in real scientific fields are exposed, not by comparing their words to their work, but by comparing the evidence to reality. > Do you realise just how kook-like your behaviour is when ever the subject approaches psychiatry? 1. Are you aware that your arguments have no content, nothing that might be mistaken for science or good-faith debate, as a result of which you're making psychology look even more ridiculous that it already does? 2. Are you aware that my views on this topic now represent the mainstream? http://www.huffingtonpost.com/dr-harold-koplewicz/dsm-mental-health-research_b_3247960.html http://www.huffingtonpost.com/dr-harold-koplewicz/dsm-mental... Ten years ago, I started a campaign against the use of Asperger Syndrome as a diagnosis, arguing that it was a particularly absurd diagnosis, able to reliably "diagnose" bright kids as mentally ill. After years of hearing from people like you who know nothing about the topic under discussion, Asperger Syndrome was abandoned for the reasons I gave. In this new effort, the DSM is being challenged for its lack of scientific rigor, and not just by me, but by people like the current chairman of the NIMH. Conclusion? You need a reality check. If psychologists were to read your replies to my posts, they would say, "Please -- don't be on our side."
  - PhasmaFelis 12y ago
    > Ten years ago, I started a campaign against the use of Asperger Syndrome as a diagnosis, arguing that it was a particularly absurd diagnosis, able to reliably "diagnose" bright kids as mentally ill. After years of hearing from people like you who know nothing about the topic under discussion, Asperger Syndrome was abandoned for the reasons I gave. Asperger's wasn't "abandoned," it was folded into the broader spectrum of autism--and not by the field in general but only by the DSM, which you seem to distrust specifically, so I'm not quite sure where you're going with this claim. My Asperger's/autism diagnosis has made a huge difference to my quality of life, by helping me understand, deal with, and find support for my particular social, sensory, and mental-processing issues. The same is true for thousands of others, and we're kind of tired of you psychology deniers telling us that we don't exist.
DanBC 12y ago
> In a real science, Diederik Stapel would have been exposed within days of entering a laboratory. You ignore the fact that scientific fraud exists in other sciences; and that this article talks about extending the use of this tool to different sciences.
- lutusp 12y ago
  > You ignore the fact that scientific fraud exists in other sciences It's not the topic under discussion, but scientific fraud certainly exists among scientists, with one important difference -- real scientists are exposed by comparing their work, their evidence, to reality. Stapel was exposed by comparing his words to his work -- work often not even conducted. There was never any serious prospect of comparing Stapel's work to reality, because his work didn't try to address reality -- that's science's purview. In science, reliant on physical evidence, how you describe the evidence shouldn't matter, the evidence should speak for itself. Reliable evidence should bring different, similarly equipped observers to the same conclusion. In psychology, the reason there's no consensus on the meaning of evidence is because the evidence is extremely poor, freeing different psychologists to draw different conclusions from the same evidence. > this article talks about extending the use of this tool to different sciences. Yes, that's true, they do talk about it. But until they actually try to apply it, it's not a legitimate topic of discussion. You haven't asked an obvious question -- where are the falsifiable psychological theories, theories that can be proven false in practical tests, but that resist falsification? Where are the time-tested theories that would make psychology one field, like physics or biology, and that would justify clinical practice?

shawn-furyan 12y ago

A 30% error rate is far, far too large for this to be useful as a first line filter. Even error rates of less than 1/3 of that (10%) lead to combinatorial effects that make for very unreliable tests. At best, this would be a corroborating test, and not a very strong one at that.

rflrob 12y ago
What I, if I were a reviewer on this paper, would have liked to have seen is a ROC curve—basically, if you tweak the cut offs, can you get fewer false positives at the expense of more false negatives. Assuming fraud in science is rare (I think even the gloom-and-doom-iest reports put it under 20% for any kind of research misconduct, not just fraud), even a moderately low false positive rate will flag a lot more papers than a perfect true positive rate.
a3n 12y ago
Think of it as early days.

rflrob 12y ago

It seems to me that every researcher will have a different level of 'amplifiers' in their writing, so this will be a really hard classifier to train generally.