4 ms·
Wouldn't changing the assumed rate of 50% to something higher make the probability less extreme? I know my initial inclination is to believe that less than 50%
by Vraxx 11y ago
Wouldn't changing the assumed rate of 50% to something higher make the probability less extreme? I know my initial inclination is to believe that less than 50% of the published studies have results that are able to be replicated, but that could be just the cynic in me. I know that the article mentioned that just 39% of the selected studies were able to replicate the results, but that could be caused by a selection bias in which 44 studies were used. I'm inclined to believe this study, especially because the results are rather amusing, but there is definitely room to be more sure.
- evmar 11y agoThank you for thinking critically about my comment! As I mentioned I am an amateur at stats and I now think I modeled the problem wrong. Let's hypothetically suppose that we knew that 75% of the studies were replicable. Can we make a better coin flip prediction? If you had a coin flip that says yes 75% of the time, it isn't necessarily correct at a rate of 75% -- instead it'd be right .75^2+.25^2 = 62.5% of the time. In fact a coin that just predicted "replicable" every time in this scenario would be right 75% of the time. So I think maybe my null hypothesis should've been based on "did they do better than a parrot that always says yes", not a coin flip. I think the math in the original problem stays the same, it's just you change it to "a coin that always predicts it will be replicable". And in that case, if the underlying rate of replicable was 71%, then their prediction market only does as well as the always-yes coin and is in fact not very useful.