3 ms·
> If experts [in prediction markets] can instinctively spot an irreproducible finding, “that kind of begs the question of why that doesn’t seem to be happening
by hirundo 8y ago
> If experts [in prediction markets] can instinctively spot an irreproducible finding, “that kind of begs the question of why that doesn’t seem to be happening in peer review,” says Fiona Fidler, a philosopher of science at The University of Melbourne in Australia. But if future studies can identify and weigh the best predictors of replicability, reviewers might be given a rubric to help them weed out problematic work before it’s published.
That's a troubling suggestion. Results are valuable to the extent that they're both accurate and surprising. To systematically suppress surprising results as a negative predictor of accuracy sounds like a formula for suppressing surprisingly valuable papers.
- JumpCrisscross 8y ago> to systematically suppress surprising results "Weeding out" doesn't mean systematically suppressing. It simply means exhibiting caution before putting the paper in a reputable journal. The Internet provides the entire system with an escape valve in scientists' capacities to publish papers on their own websites. If the finding is surprising enough, it shouldn't be problematic attracting some attention, particularly in the social sciences.
- hirundo 8y ago> "Weeding out" doesn't mean systematically suppressing. It does when I'm weeding my garden. A result that is weeded out of a journal is suppressed from taking root in the minds of its readers. I agree that extraordinary claims require extraordinary evidence to accept. But as long as they meet the ordinary standard of rigor it's just those claims that inquiring readers want to entertain.
- thanatropism 8y agoThink of Thikonov regularization. Or the LASSO. The basic idea is to start with a prior that the world is uninteresting and let the evidence try to prove otherwise.
- deleted 8y ago[deleted]
- dev_dull 8y agoI believe the thought is extraordinary results require extraordinary proof.
- closed 8y agoThe danger with surprising results is that they often use the surprise part as an excuse for small sample sizes, etc.. Another way to think about this situation is to flip it and say that surprising (and spurious) results are more likely from exploring a small dataset. Andrew Gelman talks a lot about this issue on his blog: https://andrewgelman.com/2014/08/01/scientific-surprise-two-step/ https://andrewgelman.com/2014/08/01/scientific-surprise-two-...
- deleted 8y ago[deleted]