5 ms·

tldr; Efforts to reproduce 100 psychological findings. Only 39 were reproducible. And 61 were not. Hal Pashler, “A lot of working scientists assume that if it’

tldr; Efforts to reproduce 100 psychological findings. Only 39 were reproducible. And 61 were not. Hal Pashler, “A lot of working scientists assume that if it’s published, it’s right,” he says. “This makes it hard to dismiss that there are still a lot of false positives in the literature.”

nzseth 12y ago
There seems to be two factors at play here that are creating almost nonsensical results. 1. The results, when filtered through the replicability guidelines rendered a clear verdict - 61 to 39 against. 2. The question of "how closely did the findings resemble the original study?" flips the findings. Moderately similar findings are the majority - 58 to 42 in the other direction. How can you have a study that has "virtually identical" findings that doesn't replicate the original?
- kistaro 12y ago
  Original results that were exactly on the borderline of statistical significance, with new results that were barely on the "fail" side of that line, but both sets of results are within the margin of error of each other? (I am not a frequentist, so my bias may be showing through)
innguest 12y ago
Thanks for that. That's Psychology in a nutshell for you folks, this amazing "science". Now let's wait for a similar study for economics, the other amazing "science".
- 0xdeadbeefbabe 12y ago
  Are you suggesting it's worse than food science?
  - ZeroFries 12y ago
    Where does he suggest that?
- danmaz74 12y ago
  How do you define "science"?
  - DanBC 12y ago
    Not like this "scientist" at a reputable university. http://wjh.harvard.edu/~jmitchel/writing/failed_science.htm http://wjh.harvard.edu/~jmitchel/writing/failed_science.htm > · Recent hand-wringing over failed replications in social psychology is largely pointless, because unsuccessful experiments have no meaningful scientific value. > · Because experiments can be undermined by a vast number of practical mistakes, the likeliest explanation for any failed replication will always be that the replicator bungled something along the way. Unless direct replications are conducted by flawless experimenters, nothing interesting can be learned from them. {edit: read the link. It's scary to read this stuff coming from a science PhD.} Other psychologists have realised that there's a "crisis of replication" in psychology. But it's not just psychology, it affects other sciences (albeit to varying degrees) too. http://slatestarcodex.com/2014/04/28/the-control-group-is-out-of-control/ http://slatestarcodex.com/2014/04/28/the-control-group-is-ou...
    - danmaz74 12y ago
      Ok, but one thing is to say that there are bad scientists in the field of psychology, and another one is to say psychology is not a science at all. It's not (and it can't be) an "exact" science, but it's science all the same.
  - innguest 11y ago
    A Science makes testable hypotheses that are falsifiable. Economics and Psychology don't make falsifiable predictions. When something goes wrong in either of those fields, they always have an explanation - they're never wrong. Economy didn't lift off in 2015 Q1? It was the weather. But we know the weather affects all Q1, can't they factor that into the prediction for Q1 numbers? No, because they want to be able to use the weather excuse. Follow what they say and you will start to see the pattern of excuses. Every time they are sure something will work; then it doesn't work; then they claim they knew all along it wouldn't work, but this new solution will definitely work. Rinse, repeat.
- deleted 12y ago
  [deleted]
plg 12y ago
should be pointed out that just as 100 findings doesn't mean that 100 are true, it's also the case that 61 non-reproducible doesn't mean that 61 are false
- analog31 12y ago
  It Confirms that a good rule of thumb for laypeople is to multiply the p value by ten.
hackuser 12y ago
I don't think that represents well what was said: * "Of the 61 non-replicated studies, scientists classed 24 as producing findings at least “moderately similar” to those of the original experiments, even though they did not meet pre-established criteria" * "Daniele Fanelli, who studies bias and scientific misconduct at Stanford University in California, says ... psychology does not necessarily lag behind ... other sciences. ... earlier studies have suggested that reproducibility rates in cancer biology and drug discovery could be even lower"

nzseth 12y ago

There seems to be two factors at play here that are creating almost nonsensical results. 1. The results, when filtered through the replicability guidelines rendered a clear verdict - 61 to 39 against. 2. The question of "how closely did the findings resemble the original study?" flips the findings. Moderately similar findings are the majority - 58 to 42 in the other direction. How can you have a study that has "virtually identical" findings that doesn't replicate the original?

kistaro 12y ago
Original results that were exactly on the borderline of statistical significance, with new results that were barely on the "fail" side of that line, but both sets of results are within the margin of error of each other? (I am not a frequentist, so my bias may be showing through)

innguest 12y ago

Thanks for that. That's Psychology in a nutshell for you folks, this amazing "science". Now let's wait for a similar study for economics, the other amazing "science".

0xdeadbeefbabe 12y ago
Are you suggesting it's worse than food science?
- ZeroFries 12y ago
  Where does he suggest that?
danmaz74 12y ago
How do you define "science"?
- DanBC 12y ago
  Not like this "scientist" at a reputable university. http://wjh.harvard.edu/~jmitchel/writing/failed_science.htm http://wjh.harvard.edu/~jmitchel/writing/failed_science.htm > · Recent hand-wringing over failed replications in social psychology is largely pointless, because unsuccessful experiments have no meaningful scientific value. > · Because experiments can be undermined by a vast number of practical mistakes, the likeliest explanation for any failed replication will always be that the replicator bungled something along the way. Unless direct replications are conducted by flawless experimenters, nothing interesting can be learned from them. {edit: read the link. It's scary to read this stuff coming from a science PhD.} Other psychologists have realised that there's a "crisis of replication" in psychology. But it's not just psychology, it affects other sciences (albeit to varying degrees) too. http://slatestarcodex.com/2014/04/28/the-control-group-is-out-of-control/ http://slatestarcodex.com/2014/04/28/the-control-group-is-ou...
  - danmaz74 12y ago
    Ok, but one thing is to say that there are bad scientists in the field of psychology, and another one is to say psychology is not a science at all. It's not (and it can't be) an "exact" science, but it's science all the same.
- innguest 11y ago
  A Science makes testable hypotheses that are falsifiable. Economics and Psychology don't make falsifiable predictions. When something goes wrong in either of those fields, they always have an explanation - they're never wrong. Economy didn't lift off in 2015 Q1? It was the weather. But we know the weather affects all Q1, can't they factor that into the prediction for Q1 numbers? No, because they want to be able to use the weather excuse. Follow what they say and you will start to see the pattern of excuses. Every time they are sure something will work; then it doesn't work; then they claim they knew all along it wouldn't work, but this new solution will definitely work. Rinse, repeat.
deleted 12y ago
[deleted]

plg 12y ago

should be pointed out that just as 100 findings doesn't mean that 100 are true, it's also the case that 61 non-reproducible doesn't mean that 61 are false

analog31 12y ago
It Confirms that a good rule of thumb for laypeople is to multiply the p value by ten.

hackuser 12y ago

I don't think that represents well what was said: * "Of the 61 non-replicated studies, scientists classed 24 as producing findings at least “moderately similar” to those of the original experiments, even though they did not meet pre-established criteria" * "Daniele Fanelli, who studies bias and scientific misconduct at Stanford University in California, says ... psychology does not necessarily lag behind ... other sciences. ... earlier studies have suggested that reproducibility rates in cancer biology and drug discovery could be even lower"