3 ms·

"It was surprising that even with these efforts we were only able to obtain support for the original findings for half of the studies. These results do not def

"It was surprising that even with these efforts we were only able to obtain support for the original findings for half of the studies. These results do not definitively mean that the original findings were wrong, but they do suggest that they are not as robust as might have been assumed." - Michelangelo Vianello, one of the project leads. Yes, yes they do. It's ok to say the original findings were wrong. It's ok to re-assert the null hypothesis. The unwillingness to do so undermines the process and is disconcerting to say the least.

tlb 8y ago
Failure to replicate doesn't prove the original findings were wrong. Other things that can cause replication failure are: - Poor explanation of experimental procedure, so the replication doesn't match the original experiment. - Awareness of the effect may invalidate it. For example, it'd be hard to replicate the Stanford Prison Experiment because every psychology undergrad has heard the story. - Change in social conditions or prevailing attitudes. Consumer behavior (the subject of some of these studies) has changed significantly over the decades. Homophobia (another study) varies dramatically over time and place.
- iandanforth 8y ago
  We should continue this argument over coffee, hope you're doing well!
  - skrebbel 8y ago
    Please record the audio, i'm curious who convinces who. (or just continue here on HN of course)
pfortuny 8y ago
Not exactly: it is ok to say that there were no original findings. This is a tiny semantic detail but important: if the author does not support his findings, it is OK to re-assert the null hypothesis but not to say that "the findings were wrong". What I mean is: "no experiment" does not mean the same as "wrong experiment".
- iandanforth 8y ago
  Well, let's agree that 'wrong' doesn't mean 'bad.' I use the term in the Feynman sense, if your guess doesn't agree with experiment, it's wrong. The assertion that they found a statistically significant effect was wrong (or to Trevor's point in a sibling comment, is no longer true.)
blbviviv 8y ago
They are saying that they made have mistakes now, it's rather defensive. Isn't it?

tlb 8y ago

Failure to replicate doesn't prove the original findings were wrong. Other things that can cause replication failure are: - Poor explanation of experimental procedure, so the replication doesn't match the original experiment. - Awareness of the effect may invalidate it. For example, it'd be hard to replicate the Stanford Prison Experiment because every psychology undergrad has heard the story. - Change in social conditions or prevailing attitudes. Consumer behavior (the subject of some of these studies) has changed significantly over the decades. Homophobia (another study) varies dramatically over time and place.

iandanforth 8y ago
We should continue this argument over coffee, hope you're doing well!
- skrebbel 8y ago
  Please record the audio, i'm curious who convinces who. (or just continue here on HN of course)

pfortuny 8y ago

Not exactly: it is ok to say that there were no original findings. This is a tiny semantic detail but important: if the author does not support his findings, it is OK to re-assert the null hypothesis but not to say that "the findings were wrong". What I mean is: "no experiment" does not mean the same as "wrong experiment".

iandanforth 8y ago
Well, let's agree that 'wrong' doesn't mean 'bad.' I use the term in the Feynman sense, if your guess doesn't agree with experiment, it's wrong. The assertion that they found a statistically significant effect was wrong (or to Trevor's point in a sibling comment, is no longer true.)

blbviviv 8y ago

They are saying that they made have mistakes now, it's rather defensive. Isn't it?