4 ms·
> If an initial replication attempt failed, the researchers added even more participants. This is an obvious way to tamper with the results; it's just more of
by jblow 8y ago
> If an initial replication attempt failed, the researchers added even more participants.
This is an obvious way to tamper with the results; it's just more of the same kind of p-hacking that bad researchers are so often doing. They are using "we re-do the study with a larger population" as a way to re-roll the dice if the first die roll doesn't come up the way they want. (Note that if the die roll did come up the way they want, they don't re-do the study with a larger population in order to see if the replication fails).
Nobody should be taking this seriously.
- timr 8y agoThere are known methods to account for this, and if you adjust the statistics properly to account for the repeat, it isn't p-hacking.
- jblow 8y agoDid they? I can't tell from the article!
- philipodonnell 8y agoDo they do that?
- ggggtez 8y agoI think the point is actually more the opposite. If your study failed to reproduce even with the generous "keep adding more participants until it works" method, then your first study is beyond a doubt BS.
- jblow 8y agoExcept no, because rhetorically they are using the 62% figure as a good sign, rather than a bound on how bad things are.
- gweinberg 8y agoI'd say it's a good article with a bad title. my take away is that even with their generous approach only 62% of studies replicate. That means 1) probably over 1/3 of results are completely bogus and 2) of the ones that were real, about half the effects were substantially weaker than claimed.
- SiempreViernes 8y agoThis article is Science telling us that "look, over half of the high impact science we publish is actually real! Our peer review works!".
- im3w1l 8y agoIt's not 100% unproblematic but I wouldn't call it p-hacking.
- nolemurs 8y agoRepeating a study if, and only if, the study failed to provide the desired result is the most basic p-hacking technique there is. If this isn't p-hacking, I don't know what is! To give a simple model, suppose you decide an effect is significant if there's only a 5% chance that you'd see this data if the effect didn't exist. If you run the experiment again with the same threshold for significance when don't get the desired result, then the probability of seeing an effect that doesn't exist rises to 9.75% (= 0.05 + 0.95 * 0.05). The effect isn't merely "not 100% unproblematic" it's a serious problem! You've gone from what looks like a p-value of 5% to a p-value of 9.75%. The fact that the second study is done with a larger sample is pretty much irrelevant unless it also comes with a higher p-value threshold for you to accept the result.