3 ms·
I think the most damning thing to come out of the replication crisis was when they asked a bunch of scientists to place bets on whether a given paper (with p <
by Analemma_ 9y ago
I think the most damning thing to come out of the replication crisis was when they asked a bunch of scientists to place bets on whether a given paper (with p < 0.05) would replicate, and it turned out these bets were right quite often (https://fivethirtyeight.com/features/how-to-tell-good-studies-from-bad-bet-on-them/ https://fivethirtyeight.com/features/how-to-tell-good-studie...).
That shouldn't be possible! Science is supposed to be the best possible epistemological methodology, and here it is being beaten in "success rate of determining true from false" by guessing. What's immensely frustrating is that it's not a question of whether we're just not smart enough to tell true from false, we clearly have the power (since the guesses were often right) but we're not using it. Whatever "truth compass" the guessers were using should be part of the scientific process somehow. That's what is "broken".
- whatshisface 9y agoThat "truth compass" they used was reading the method section, which is far from guessing and very close to the scientific process.
- Analemma_ 9y agoRight, I'm not chalking it up to mysterious psychic powers. What I mean is, it's frustrating in how it demonstrates that our truth-sensing capabilities, and the "pure" scientific method, are actually working just fine and we don't need to have any existential panic over whether truth is unknowable even in noisy data. But those capabilities aren't being used when it comes time to actually write the paper, which is exactly when you should be applying them.
- LeifCarrotson 9y agoIt shouldn't be, though. It may be easy for a scientist to look at the method sections of two papers that had p<0.05 results and say of the first "that looks like a solid study with a reasonable conclusion and robust controls" and of the other "that's bogus". The problem is that without this expert panel making these calls, both studies got published, were given the same credibility, and got consumed by non-experts. Peer review and the publishing process are supposed to weed out the obviously bad studies, but they're not doing so.
- haihaibye 9y agoScientists are familiar with the literature and are taking into account information from other papers when judging a particular one. Individual papers are not a source of truth, they're bits of cumulative evidence hopefully leading to the truth.
- mattnewton 9y agoI don't understand, isn't that really encouraging, that peer review might work to weed out the truth after all?
- LeifCarrotson 9y agoPeer review is supposed to be done before publishing, not in a separate meta-analysis.