3 ms·
"Replication studies: Bad copy" is very interesting. Sample quote: “I've seen students spending their entire PhD period trying to replicate a phenomenon, failin
by bambax 14y ago
"Replication studies: Bad copy" is very interesting. Sample quote: “I've seen students spending their entire PhD period trying to replicate a phenomenon, failing, and quitting academia because they had nothing to show for their time.”
We say that a study "succeeds" when it illustrates a novel behavior and "fails" when the expected behavior doesn't happen.
This is a very slippery use of words.
A finding that can't be replicated isn't a finding. It's a rumor, an urban myth, but not a scientific step forward.
A study that "fails to replicate" a previous finding isn't a "failure" -- it succeeded at debunking a false belief! And it should be granted wide access to publishing.
(Same thing happens in medical studies BTW, where it's arguably much more dangerous).
- chii 14y ago> A study that "fails to replicate" a previous finding isn't a "failure". Yep. the problem is that to the "laymen", it seems a bunch of people wasted time trying to re-do what someone else has "already" done, and failed. Whether the failure was due to incompetance, chance, or an actual failure to replicate (meaning it is a significant), to the mind of the laymen, they are one and the same. this then end up being difficult for people who might fund such studies to fund these sorts of studies (despite it being quite useful), and leads to people seeking "newer" things to research. its a bit of a chilling effect really.
- StavrosK 14y agoWho are these laymen, and why are they reviewing papers?
- chii 14y agothey are not reviewing papers, nor are they the fund contributors. They are the general population, whose sentiments affect the way society sees things overall. For example, a lot of people are now rightfully scared of nuclear tech (due to recent disasters). And yet, most people dont understand that neclear energy is quite safe (compared to coal, say). If a political leader decides to invest in nuclear energy, they instantly become unpopular, and thus lies the problem of funding via such channels.
- StavrosK 14y agoSure, but they aren't the ones deciding what gets published or not. The study that tried to replicate another study but failed has already been done, and it's not up to the laymen to decide whether or not to publish the results. Nor can we predict which studies will fail to duplicate results and not fund them, otherwise we wouldn't have to make them. Results of failure are still results, and they represent valuable knowledge. Too bad the people who got them consider them a failure.
- lars512 14y agoIf there's an obvious flaw in the existing study, you can make something out of a new study which doesn't contain that flaw. But it's very hard to convince yourself (and others) that replicating/failing to replicate someone's work is the same level of contribution as the original work. What would be great: a centralised place for people to share their attempts to replicate a work. If several groups have failed, and nobody has succeeded, then there's certainly room for a rigorous, joint debunking of the experiment. If, as is often the case, there's some subtlety to get the experiment right, then the discussion around it between groups actually enhances the original work.
- lutusp 14y ago> If there's an obvious flaw in the existing study, you can make something out of a new study which doesn't contain that flaw. Not necessarily. How would you repair a study that requires strict controls but human subjects, without violating ethical constraints? You can't. Some studies are perpetually hobbled by the fact that you can't put human subjects in rat cages and force-feed them. My point is that, beyond the replicability issues discussed on the linked article, there are serious and ongoing obstacles to rigorous experimental work in psychology.
- bambax 14y ago> Results of failure are still results Yes! I would add that failing to replicate existing results is the only interesting outcome of a replicating study. If we try to replicate a study and arrive at the same conclusions, we have learned little (the original result is a little stronger but that's about it). If however, a series of new studies fail to replicate original results, we have learned a lot -- we have learned that those original results were false. I think the whole problem is, ironically, psychological in nature. Results of studies become articles of faith; people identify with their beliefs and hate to admit they have been fooled...