4 ms·

If we report p values around 0.05 wouldn’t you expect about 1 in 20 studies failing to replicate without being bad science?

zhdc1 5y ago
In the social sciences? If they're reproducing the earlier results with the original data and code, the results should be exactly the same. If they're replicating with a follow-on study of some kind, it depends on any number of factors, since it's almost impossible to do a follow-on survey-based or lab experiment or use a similar or newer dataset while accounting for and controlling exactly the same set of conditions (this is one of the issues that I've heard used to distinguish between hard and soft sciences). E.g., if paper A has an observation count of 200 (insert your preferred number), a replication study could hand out exactly the same survey to another 200 respondents and get similar results that, nonetheless, fail to exactly replicate the earlier findings (or, over a small number of samples, fail to conform to a normal distribution). This doesn't mean that the original study was "bad science" or that the social sciences in general are "bad", only that we have to temper our expectations and maintain a healthy dose of skepticism.
MereInterest 5y ago
Not quite. That's the common misinterpretation of p-values as probabilities that a study is correct. The p-value isn't the probability of the hypothesis being wrong given the data. It is instead the probability of the data given the hypothesis being wrong. In order to determine what proportion of studies would fail to reproduce, you would also need to know the number of correct hypotheses in the first place. If 100 correct hypotheses and 900 incorrect hypotheses are tested, your sample of "p<0.05" results would contain 95 correct hypotheses and 45 incorrect hypotheses, so about 1/3 of the results would fail to replicate. This is why there is such a large push for prepublishing of analysis methods, hypotheses, and experimental design, and later publishing of negative results. The initial distribution of hypotheses is important to know how to interpret the resulting body of work.
deleted 5y ago
[deleted]

zhdc1 5y ago

In the social sciences? If they're reproducing the earlier results with the original data and code, the results should be exactly the same. If they're replicating with a follow-on study of some kind, it depends on any number of factors, since it's almost impossible to do a follow-on survey-based or lab experiment or use a similar or newer dataset while accounting for and controlling exactly the same set of conditions (this is one of the issues that I've heard used to distinguish between hard and soft sciences). E.g., if paper A has an observation count of 200 (insert your preferred number), a replication study could hand out exactly the same survey to another 200 respondents and get similar results that, nonetheless, fail to exactly replicate the earlier findings (or, over a small number of samples, fail to conform to a normal distribution). This doesn't mean that the original study was "bad science" or that the social sciences in general are "bad", only that we have to temper our expectations and maintain a healthy dose of skepticism.

MereInterest 5y ago

Not quite. That's the common misinterpretation of p-values as probabilities that a study is correct. The p-value isn't the probability of the hypothesis being wrong given the data. It is instead the probability of the data given the hypothesis being wrong. In order to determine what proportion of studies would fail to reproduce, you would also need to know the number of correct hypotheses in the first place. If 100 correct hypotheses and 900 incorrect hypotheses are tested, your sample of "p<0.05" results would contain 95 correct hypotheses and 45 incorrect hypotheses, so about 1/3 of the results would fail to replicate. This is why there is such a large push for prepublishing of analysis methods, hypotheses, and experimental design, and later publishing of negative results. The initial distribution of hypotheses is important to know how to interpret the resulting body of work.

deleted 5y ago

[deleted]