3 ms·
> Researchers’ expertise, prior beliefs, and expectations barely predict the wide variation in research outcomes. More than 90% of the total varian
by SamBam 4y ago
> Researchers’ expertise, prior beliefs, and expectations barely predict the wide variation in research outcomes. More than 90% of the total variance in numerical results remains unexplained even after accounting for research decisions [including which statistical tests to perform] identified via qualitative coding of each team’s workflow.
They're saying that scientists, given the same data, will achieve widely different results, and that they can't even work out how they're getting these different results. That's surprising and concerning.
> This reveals a universe of uncertainty that is otherwise hidden when considering a single study in isolation. The idiosyncratic nature of how researchers’ results and conclusions varied is a new explanation for why many scientific hypotheses remain contested.
- Enginerrrd 4y agoI've certainly got into the habit of reading the methods and results of social science papers that make headlines and what I've found has been horrifying. More than half of the time, there are serious, serious flaws involved in the data collection or the analysis. Even worse, I've frequently encountered situations where the underlying data really seems to point toward the opposite conclusion than the one made by the researchers who have really twisted and beaten the data into submission to get the conclusion they wanted.
- DontchaKnowit 4y agoYep, which is why I find the use of "science" as a political battering ram to be one of the most infuriating modern phenomena.
- jltsiren 4y agoIt's not surprising at all. There are always many arbitrary choices to make and many things that can go wrong if you try to make conclusions from data. Even worse if you try to automate making the conclusions. And because the data you have has often been derived from lower-level data, many arbitrary choices have already been made and many things have already gone wrong. On good days, I find it a small miracle that we can sometimes make justified conclusions from data. On bad days, it feels that making conclusions from data is storytelling, not science.
- a1369209993 4y ago> On good days, I find it a small miracle that we can sometimes make justified conclusions from data. On bad days, it feels that making conclusions from data is storytelling, not science. If you dig far enough down the rabbithole, you eventually find that the numbers are being pulled out of someone's proverbial ass. But pulling numbers out of your ass and using them to make a conclusion is a lot better than pulling a conclusion out of your ass and using it to make numbers.