Thank you for thinking critically about my comment! As I mentioned I am an amateur at stats and I now think I modeled the problem wrong. Let's hypothetically

Thank you for thinking critically about my comment! As I mentioned I am an amateur at stats and I now think I modeled the problem wrong. Let's hypothetically suppose that we knew that 75% of the studies were replicable. Can we make a better coin flip prediction? If you had a coin flip that says yes 75% of the time, it isn't necessarily correct at a rate of 75% -- instead it'd be right .75^2+.25^2 = 62.5% of the time. In fact a coin that just predicted "replicable" every time in this scenario would be right 75% of the time. So I think maybe my null hypothesis should've been based on "did they do better than a parrot that always says yes", not a coin flip. I think the math in the original problem stays the same, it's just you change it to "a coin that always predicts it will be replicable". And in that case, if the underlying rate of replicable was 71%, then their prediction market only does as well as the always-yes coin and is in fact not very useful.