3 ms·

> The 0.05 threshold is indeed arbitrary, but the scientific method is sound. Agreed. A single published paper is not science, a tree data structure of publis

by qudat 2y ago

> The 0.05 threshold is indeed arbitrary, but the scientific method is sound. Agreed. A single published paper is not science, a tree data structure of published papers that all build off of each other is science.

schmidtleonard 2y ago
Right, the decisions made by future researchers about what to base their work on are the real evaluation, hence citation counts as a core metric. It's easy to claim positive results by making your null hypothesis dogshit (and choice of "p" is easily the least inspired way to sabotage a null hypothesis), but researchers learn this game early and tend to not gamble their time following papers where they suspect this is what's going on. The whole thing kinda works, in a world where the alternatives don't work at all.
greentxt 2y ago
Sounds good but is that true? A single unreplicated paper could be science couldn't it? Science is a framework within which there are many things, including theories, mistakes, false negatives, replication failures, etc... Science progresses due to quantity more than quality, it is brute force in some sense that way, but it is more a journey than a destination. You "do" science moreso than you "have" science.
- bluGill 2y ago
  Only scientists care about doing science. Most people are not scientists. Even scientists are not scientists in every field. We as the genereral population (including scientists in a different field) however care about science because of the results. The results of science is modern health care, engineering (bridges that don't collapse...), and many other such things that we get because we "have" science.
- bunderbunder 2y ago
  A single brick on the ground, all by itself, is not a wall. But if you take a lot of bricks and arrange them appropriately, then every single one of those bricks is wall. In other words, just like the article points out down in the "dos" section, it depends on how you're treating that single unreplicated paper. Are you cherry-picking it, looking at it in isolation, and treating it as if it were definitive all by itself? Or are you considering it within a broader context of prior and related work, and thinking carefully about the strengths, limitations, and possible lacunae of the work it represents?
- jhbadger 2y ago
  I think you and the OP are agreeing with each other. The issue with a "single unreplicated paper" is exactly the issue you bring up with science as a journey. It's possible that this paper has found a genuine finding or that it is nonsense (people can find isolated published papers supporting almost anything they want even if they don't reflect the scientific consensus), but if no other researchers are even bothering to replicate the findings in it it hasn't joined the journey.
zmgsabst 2y ago
Only if there isn’t systemic compromise via funding allocation, eg, what happened with Alzheimers research.
refurb 2y ago
Precisely. As a scientist, that’s how it works. If a new paper with an outrageous claim pops up, people are automatically suspicious. Until it’s been reproduced by a few labs, it’s just “interesting”. Then once it’s been validated and new science is built off of it, it’s not really accepted as foundational.

by qudat 2y ago

schmidtleonard 2y ago

Right, the decisions made by future researchers about what to base their work on are the real evaluation, hence citation counts as a core metric. It's easy to claim positive results by making your null hypothesis dogshit (and choice of "p" is easily the least inspired way to sabotage a null hypothesis), but researchers learn this game early and tend to not gamble their time following papers where they suspect this is what's going on. The whole thing kinda works, in a world where the alternatives don't work at all.

greentxt 2y ago

Sounds good but is that true? A single unreplicated paper could be science couldn't it? Science is a framework within which there are many things, including theories, mistakes, false negatives, replication failures, etc... Science progresses due to quantity more than quality, it is brute force in some sense that way, but it is more a journey than a destination. You "do" science moreso than you "have" science.

bluGill 2y ago
Only scientists care about doing science. Most people are not scientists. Even scientists are not scientists in every field. We as the genereral population (including scientists in a different field) however care about science because of the results. The results of science is modern health care, engineering (bridges that don't collapse...), and many other such things that we get because we "have" science.
bunderbunder 2y ago
A single brick on the ground, all by itself, is not a wall. But if you take a lot of bricks and arrange them appropriately, then every single one of those bricks is wall. In other words, just like the article points out down in the "dos" section, it depends on how you're treating that single unreplicated paper. Are you cherry-picking it, looking at it in isolation, and treating it as if it were definitive all by itself? Or are you considering it within a broader context of prior and related work, and thinking carefully about the strengths, limitations, and possible lacunae of the work it represents?
jhbadger 2y ago
I think you and the OP are agreeing with each other. The issue with a "single unreplicated paper" is exactly the issue you bring up with science as a journey. It's possible that this paper has found a genuine finding or that it is nonsense (people can find isolated published papers supporting almost anything they want even if they don't reflect the scientific consensus), but if no other researchers are even bothering to replicate the findings in it it hasn't joined the journey.

zmgsabst 2y ago

Only if there isn’t systemic compromise via funding allocation, eg, what happened with Alzheimers research.

refurb 2y ago

Precisely. As a scientist, that’s how it works. If a new paper with an outrageous claim pops up, people are automatically suspicious. Until it’s been reproduced by a few labs, it’s just “interesting”. Then once it’s been validated and new science is built off of it, it’s not really accepted as foundational.