4 ms·

Sure, but lots of high profile papers in top end journals like Nature fail to replicate: https://www.vox.com/science-and-health/2018/8/27/17761466/psychology-re

Sure, but lots of high profile papers in top end journals like Nature fail to replicate: https://www.vox.com/science-and-health/2018/8/27/17761466/psychology-replication-crisis-nature-social-science https://www.vox.com/science-and-health/2018/8/27/17761466/ps... If we are to learn anything from the last 10 years of the replication crisis, it has to be that a single paper is a data point, not a conclusion.

Fishysoup 6y ago
Scientists, especially in biology (possibly psychology too but i don't know), do look at papers as data points and not conclusions. The "conclusion-minded" treatment of papers is a sad artefact of the publish-or-perish system (have to add a ton of spin to your results to make them publishable) and of pop science journalism.
DoingIsLearning 6y ago
This is neither new nor surprising. There is a whole field of science dedicated to the statistical analysis of multiple studies in prior literature, e.g. just search for _Meta-analysis_ in the context of medical research. A state-of-the-art Meta-Analysis is indeed the strongest level of scientific evidence you can have to advise a healthcare decisions or guidelines. The only purpose of these journals with what they call a high 'impact factor', is that they typically have their peer reviews performed by people who are regarded as the _best_ in the their particular field. Nevertheless, to assume that an expert cannot be conned is also naive. edit: typo
meowface 6y ago
>If we are to learn anything from the last 10 years of the replication crisis, it has to be that a single paper is a data point, not a conclusion. I highly recommend this elaboration upon that idea: https://slatestarcodex.com/2014/12/12/beware-the-man-of-one-study/ https://slatestarcodex.com/2014/12/12/beware-the-man-of-one-...

4 ms·

Sure, but lots of high profile papers in top end journals like Nature fail to replicate: https://www.vox.com/science-and-health/2018/8/27/17761466/psychology-re

by sbierwagen 6y ago

Fishysoup 6y ago
Scientists, especially in biology (possibly psychology too but i don't know), do look at papers as data points and not conclusions. The "conclusion-minded" treatment of papers is a sad artefact of the publish-or-perish system (have to add a ton of spin to your results to make them publishable) and of pop science journalism.
DoingIsLearning 6y ago
This is neither new nor surprising. There is a whole field of science dedicated to the statistical analysis of multiple studies in prior literature, e.g. just search for _Meta-analysis_ in the context of medical research. A state-of-the-art Meta-Analysis is indeed the strongest level of scientific evidence you can have to advise a healthcare decisions or guidelines. The only purpose of these journals with what they call a high 'impact factor', is that they typically have their peer reviews performed by people who are regarded as the _best_ in the their particular field. Nevertheless, to assume that an expert cannot be conned is also naive. edit: typo
meowface 6y ago
>If we are to learn anything from the last 10 years of the replication crisis, it has to be that a single paper is a data point, not a conclusion. I highly recommend this elaboration upon that idea: https://slatestarcodex.com/2014/12/12/beware-the-man-of-one-study/ https://slatestarcodex.com/2014/12/12/beware-the-man-of-one-...