3 ms·
"For any arbitrary paper, what is your assumption about its accuracy? How much do you rely on it? Can you put a number to it? The null hypothesis that research
by Fomite 8y ago
"For any arbitrary paper, what is your assumption about its accuracy? How much do you rely on it? Can you put a number to it? The null hypothesis that research papers, especially in very difficult fields like social science, are near-infallible seems to be the error, AFAICT. It seems like something people outside science would assume, but I see scientists who are surprised by a 62% replication rate. I'm not surprised or concerned, but maybe I should be."
There is no such thing as "an arbitrary paper". It will depend very much on the study in question - including soft factors like who wrote it, but also the study design, sample size, if I think they approached the statistics appropriately, etc.
"Why do they call increasing the number of participants, "generous"? Doesn't that increase accuracy, and isn't accuracy the whole point of the replication study? Generosity implies some sort of favor, something above and beyond, while in this case it seems necessary based on the fact that the results changed - it would be failure of the replication study to not increase the number of participants."
The reason it might be thought of as generous is the criteria they determine for saying something is replicated is a significant effect in the same direction as the original study (for the record, I hate this criterion). A larger study is more likely to find a significant result if indeed there is one there, so they're giving studies that report an effect a very strong chance of seeing that replicated if indeed it was real.