4 ms·

It is funny they were worried about whether they just got lucky with their result, then did the prediction market thing, and then didn't worry whether they just

by evmar 11y ago

It is funny they were worried about whether they just got lucky with their result, then did the prediction market thing, and then didn't worry whether they just got lucky with that result! (At least the article didn't, perhaps the researchers did.) So here are some amateur stats, please check my work. This article says that the prediction market correctly predicted 71% of the replication results of 44 studies, or 31 correct. Assume the studies have a 50% chance of being replicable. Then a random coin would predict a mean of 22 correct with a std dev of sqrt(0.5 * 0.5 * 44) = 3.3. This sample has a z score of 2.72, which means there's a probability of 0.003264 (0.3%) of the random chance approach being correct 71% or better. So the result seems pretty significant. (Changing the assumed 50% to other values makes the probability even more extreme.)

socrates1998 11y ago
That's exactly what I was thinking. Is their study of "using betting to accurately predict whether studies are reproducible" also reproducible? A very interesting question about stats and studies. Even the p-value of 0.01 seems SUPER strong, but in reality, falls apart in many cases.
Vraxx 11y ago
Wouldn't changing the assumed rate of 50% to something higher make the probability less extreme? I know my initial inclination is to believe that less than 50% of the published studies have results that are able to be replicated, but that could be just the cynic in me. I know that the article mentioned that just 39% of the selected studies were able to replicate the results, but that could be caused by a selection bias in which 44 studies were used. I'm inclined to believe this study, especially because the results are rather amusing, but there is definitely room to be more sure.
- evmar 11y ago
  Thank you for thinking critically about my comment! As I mentioned I am an amateur at stats and I now think I modeled the problem wrong. Let's hypothetically suppose that we knew that 75% of the studies were replicable. Can we make a better coin flip prediction? If you had a coin flip that says yes 75% of the time, it isn't necessarily correct at a rate of 75% -- instead it'd be right .75^2+.25^2 = 62.5% of the time. In fact a coin that just predicted "replicable" every time in this scenario would be right 75% of the time. So I think maybe my null hypothesis should've been based on "did they do better than a parrot that always says yes", not a coin flip. I think the math in the original problem stays the same, it's just you change it to "a coin that always predicts it will be replicable". And in that case, if the underlying rate of replicable was 71%, then their prediction market only does as well as the always-yes coin and is in fact not very useful.
cortesoft 11y ago
Yeah, and one other thing they seem to be leaving out is the possibility that the initial measured effect IS legitimate, and it was the VERIFICATION test that produced the false result - a false negative on the verification test.

by evmar 11y ago

socrates1998 11y ago

That's exactly what I was thinking. Is their study of "using betting to accurately predict whether studies are reproducible" also reproducible? A very interesting question about stats and studies. Even the p-value of 0.01 seems SUPER strong, but in reality, falls apart in many cases.

Vraxx 11y ago

Wouldn't changing the assumed rate of 50% to something higher make the probability less extreme? I know my initial inclination is to believe that less than 50% of the published studies have results that are able to be replicated, but that could be just the cynic in me. I know that the article mentioned that just 39% of the selected studies were able to replicate the results, but that could be caused by a selection bias in which 44 studies were used. I'm inclined to believe this study, especially because the results are rather amusing, but there is definitely room to be more sure.

evmar 11y ago
Thank you for thinking critically about my comment! As I mentioned I am an amateur at stats and I now think I modeled the problem wrong. Let's hypothetically suppose that we knew that 75% of the studies were replicable. Can we make a better coin flip prediction? If you had a coin flip that says yes 75% of the time, it isn't necessarily correct at a rate of 75% -- instead it'd be right .75^2+.25^2 = 62.5% of the time. In fact a coin that just predicted "replicable" every time in this scenario would be right 75% of the time. So I think maybe my null hypothesis should've been based on "did they do better than a parrot that always says yes", not a coin flip. I think the math in the original problem stays the same, it's just you change it to "a coin that always predicts it will be replicable". And in that case, if the underlying rate of replicable was 71%, then their prediction market only does as well as the always-yes coin and is in fact not very useful.

cortesoft 11y ago

Yeah, and one other thing they seem to be leaving out is the possibility that the initial measured effect IS legitimate, and it was the VERIFICATION test that produced the false result - a false negative on the verification test.