4 ms·

> A week later Sally flips a quarter 5 times and gets 2 heads and 3 tails. She sends the results of her study to the AJCFS noting a failure to reproduce Robert'

> A week later Sally flips a quarter 5 times and gets 2 heads and 3 tails. She sends the results of her study to the AJCFS noting a failure to reproduce Robert's result, but her study is rejected because it has P=.3125. Sally must be working using the wrong null hypothesis. If she attempts to reproduce Robert's results her null hypothesis should be whatever is the one he formulated. Perhaps that the probability of tails is at most 15%. Her p-value would be 2.66% (prob of seeing <= 2 heads) and if the world was fair, AJCFS would publish her study. But the world isn't and if Robert is a renowned scientist, significant+ is enough for him but failed replications of his work requires significant+++.

kerkeslager 7y ago
I had to think about this for a while to come up with a reasonable response. What you're saying seems a bit problematic to me. The most obvious criticism that stands out to me is that when a control group is possible, that's where your null hypothesis should come from. But in the contrived coin-flipping example, there isn't a natural control, and in practice, many times control groups aren't available. So I don't know if that criticism actually applies to this situation. The next thought I have is that if your goal is to determine significance (which should be the goal), rather than to find a significant result (which should not be the goal), using the same alternative and null hypotheses makes the math for aggregating the results easier. The power of replication is that it's multiplicative: you can get an aggregate P by multiplying the P values of aggregate studies. This means a P=.2 study can become significant if it's replicated (P=.2.2=.04), while a P=.03125 study can become lose significance with just one failure to reproduce (aggregating Robert and Sally's results, P=.03125.3125^-1=.1). I guess swapping the null and alternative hypotheses is a viable strategy for bringing attention to failures to replicate and it's not mathematically invalid, but that seems like changing your hypothesis to fix a problem with the academic publisher, and it complicates aggregating the P-values of replication studies. Finally, it feels weird to me to use Robert's hypothesis as your null hypothesis. When Robert performed his experiment, his hypothesis was a stab in the dark due to a lack of pre-existing data (since he ostensibly didn't know about the other coin flipping experiments). But when attempting to reproduce a result, you have data, so it makes more sense to base your null hypothesis off the results of the previous experiment, not off the hypothesis of the previous experiment. As such, since Robert had 5 heads, the hypothesis would be that the probability of getting tails on a single coin flip is P=0, which gives Sally a VERY significant result. The numbers are weird because the experiment is contrived, but I think the general concept holds.

kerkeslager 7y ago

I had to think about this for a while to come up with a reasonable response. What you're saying seems a bit problematic to me. The most obvious criticism that stands out to me is that when a control group is possible, that's where your null hypothesis should come from. But in the contrived coin-flipping example, there isn't a natural control, and in practice, many times control groups aren't available. So I don't know if that criticism actually applies to this situation. The next thought I have is that if your goal is to determine significance (which should be the goal), rather than to find a significant result (which should not be the goal), using the same alternative and null hypotheses makes the math for aggregating the results easier. The power of replication is that it's multiplicative: you can get an aggregate P by multiplying the P values of aggregate studies. This means a P=.2 study can become significant if it's replicated (P=.2.2=.04), while a P=.03125 study can become lose significance with just one failure to reproduce (aggregating Robert and Sally's results, P=.03125.3125^-1=.1). I guess swapping the null and alternative hypotheses is a viable strategy for bringing attention to failures to replicate and it's not mathematically invalid, but that seems like changing your hypothesis to fix a problem with the academic publisher, and it complicates aggregating the P-values of replication studies. Finally, it feels weird to me to use Robert's hypothesis as your null hypothesis. When Robert performed his experiment, his hypothesis was a stab in the dark due to a lack of pre-existing data (since he ostensibly didn't know about the other coin flipping experiments). But when attempting to reproduce a result, you have data, so it makes more sense to base your null hypothesis off the results of the previous experiment, not off the hypothesis of the previous experiment. As such, since Robert had 5 heads, the hypothesis would be that the probability of getting tails on a single coin flip is P=0, which gives Sally a VERY significant result. The numbers are weird because the experiment is contrived, but I think the general concept holds.