4 ms·
I guess it's debatable. You could consider a single experimental design, administered by many different labs across many different countries, to be "many replic
by stdbrouw 12y ago
I guess it's debatable. You could consider a single experimental design, administered by many different labs across many different countries, to be "many replications", or you could argue that by using the exact same questionnaire and keeping other conditions as comparable as possible, it's just one big geographically dispersed replication.
(If you look at http://www.talyarkoni.org/blog/wp-content/uploads/2013/12/manylabsresults.png http://www.talyarkoni.org/blog/wp-content/uploads/2013/12/ma... you see that the results for the 3 "failed" experiments did see some effect at some labs. It's only when tallying up all the results that they had to conclude that replication had failed for these three.)
This isn't just semantics, though. For example, if one question on the questionnaire influences how survey takers answered subsequent questions, then we might well fail to replicate the results of an older study. You might then conclude that the older study isn't as generalizable as you thought it would be and that you need more research to figure out when exactly the effect occurs or not. You might even say that it's likely that the original study was just a fluke. But it doesn't mean the original hypothesis has been definitely and unequivocally refuted.
- darkxanthos 12y agoTo the idea of replicating the experiment, replicating it as closely as possible is exactly what the statistical methods assume. The other concept you're speaking to seems more like establishing causation or just generally understanding when the observed effects do or do not result. That seems better as a separate study altogether.
- stdbrouw 12y agoYou're right. I was referring specifically to the fact that in this replication study, they've actually bunched together many different studies into a single questionnaire that tests them all. So it's not a straight-up replication, but then, in the social sciences replications almost never are.