3 ms·

Unknown Knowns

• barry-cotter 8y ago
  It seems that scientists had a very good idea of which psychology studies would and would not replicate, of a sample of high impact, highly cited studies. Both a simple survey method and a prediction market identified the studies that didn’t replicate though the prediction market was better at the rank order and effect sizes. This suggests that which studies are bullshit is widely distributed knowledge. It also very strongly suggests that something as simple and cheap as surveys are powerful tools for making common knowledge which results and experiments are probably crap. As individuals at the moment the scientific community knows which experiments are crap. Surveys or prediction markets would make it common knowledge, would ensure that everyone knows everyone knows which results are crap.
  • Eridrus 8y ago
    So I took a version of that survey that told you how you were doing as you went along myself to see how it would go. And the main component to predicting correctly was really some form of occam's razor/causal reasoning, where the studies that seemed to make more intuitive sense, or you could imagine a causal mechanism, were the ones more likely to replicate. But this actually doesn't say anything about the quality of the actual study, it just shows that we should probably ask for more statistical rigor/power when the results are unexpected, because we expect the base rate of surprising findings to be lower in general.