3 ms·

Is it even clear that it would be worth it? Or do you propose a drastic reduction in papers to go with it, i.e., only things valuable enough to have someone rep

Is it even clear that it would be worth it? Or do you propose a drastic reduction in papers to go with it, i.e., only things valuable enough to have someone replicate it are published?

civilitty 3y ago
If research is not worth replicating, then how could it be anything but a useless contribution meant only to bolster someone’s publish or perish career? It just seems that the entire scientific publishing industry is there to support a jobs and prestige program. Any science is just a side effect that somehow justifies the whole racket, from NSF budget to postdoc dinner table.
- RandomLensman 3y ago
  Not strictly true: it only needs to matter not enough at the time. But, yes, arguably a fair amount of publications are largely irrelevant beyond a few people. Societies fund many things for many reasons, not sure science is worth singling out here.
  - medstrom 3y ago
    I don't mind the funding, but inclusion in a journal dresses it up with the blessing of a Scientific Result if the news ever cite it. We could have a separate kind of journal for not-yet-reproduced results, and somehow ensure that the prestige is zero (or equivalent to just posting the study on a blog).
    - RandomLensman 3y ago
      We could, but I doubt it would stop the news from picking stuff up. They often go with press releases anyway, not a journal citation. You would need incentives on the news side then to stop that.
- wredue 3y ago
  Reproducibility in some fields are difficult/impossible for a lot of reasons (funding, candidates, etc). Suppressing publishing at the same time as actively trashing the ability to even study seems like a recipe for disaster way above publishing bad papers to me.
  - naasking 3y ago
    > Reproducibility in some fields are difficult/impossible for a lot of reasons (funding, candidates, etc) It's not impossible if we're actually interested in the truth. > Suppressing publishing at the same time as actively trashing the ability to even study seems like a recipe for disaster way above publishing bad papers to me. It's not clear that a study that cannot be replicated is worth the paper its printed on in the current environment where replication rates are 50% or less across the board. Other strategies that change how we approach individual studies are not as onerous (preregistration, open data), so maybe in that world individual studies would be worth it, but even in that world replication is the only sure fire way to validate results.
    - RandomLensman 3y ago
      Not being replicated isn't the same as not being replicable. There might simply be no interest at the time. But that is no the same as the results not being right or even valuable at some point. Replication is not a be all and end all, because things could replicate even though understanding is wrong (i.e., observation is right in the publication but everything else is wrong). Even the replication itself can be wrong, if certain materials share on unknown contaminant, for example. It addresses some issues, but not all. For some areas it also not clear what replication means. What is it for theology, for example?
      - naasking 3y ago
        > But that is no the same as the results not being right or even valuable at some point. I can't imagine anyone who truly understands the current replication failure rates thinking that a single non-replicated study is valuable for anything other than informing what replications should be attempted. Just think about it: replication rate is generally less than 50%. That means you'll have a better chance of determining the truth on any question posed by a study by flipping a coin than by actually reading the study. > Replication is not a be all and end all, because things could replicate even though understanding is wrong (i.e., observation is right in the publication but everything else is wrong). The observations are the only things that we have to get right. Interpretations can be subject to decades of debate (some QM debates are ongoing a century later), but if the data isn't reliable then you're just wasting time debating falsehoods. Replications are critical to ensuring we have reliable empirical data. > Even the replication itself can be wrong, if certain materials share on unknown contaminant, for example. Indeed, and literally the only way to figure out that such variables exist and are affecting the results are by independent replications. The more replications, the better. One replication is a bare minimum threshold to demonstrate that the process of gathering the data is at least repeatable, in principle. > For some areas it also not clear what replication means. What is it for theology, for example? It makes perfect sense in any context where you're gathering empirical data. For instance, if you're surveying people's interpretation of "free will" [1], then the process via which you probe their views should be replicable. This means being clear about the specific phrasing of the questions asked, the environment in which they were asked, the makeup of the cohort, and so on. [1] https://www.researchgate.net/publication/274892120_Why_Compatibilist_Intuitions_are_not_Mistaken_A_Reply_to_Feltz_and_Millan https://www.researchgate.net/publication/274892120_Why_Compa...

Is it even clear that it would be worth it? Or do you propose a drastic reduction in papers to go with it, i.e., only things valuable enough to have someone replicate it are published?

civilitty 3y ago

If research is not worth replicating, then how could it be anything but a useless contribution meant only to bolster someone’s publish or perish career? It just seems that the entire scientific publishing industry is there to support a jobs and prestige program. Any science is just a side effect that somehow justifies the whole racket, from NSF budget to postdoc dinner table.

RandomLensman 3y ago
Not strictly true: it only needs to matter not enough at the time. But, yes, arguably a fair amount of publications are largely irrelevant beyond a few people. Societies fund many things for many reasons, not sure science is worth singling out here.
- medstrom 3y ago
  I don't mind the funding, but inclusion in a journal dresses it up with the blessing of a Scientific Result if the news ever cite it. We could have a separate kind of journal for not-yet-reproduced results, and somehow ensure that the prestige is zero (or equivalent to just posting the study on a blog).
  - RandomLensman 3y ago
    We could, but I doubt it would stop the news from picking stuff up. They often go with press releases anyway, not a journal citation. You would need incentives on the news side then to stop that.
wredue 3y ago
Reproducibility in some fields are difficult/impossible for a lot of reasons (funding, candidates, etc). Suppressing publishing at the same time as actively trashing the ability to even study seems like a recipe for disaster way above publishing bad papers to me.
- naasking 3y ago
  > Reproducibility in some fields are difficult/impossible for a lot of reasons (funding, candidates, etc) It's not impossible if we're actually interested in the truth. > Suppressing publishing at the same time as actively trashing the ability to even study seems like a recipe for disaster way above publishing bad papers to me. It's not clear that a study that cannot be replicated is worth the paper its printed on in the current environment where replication rates are 50% or less across the board. Other strategies that change how we approach individual studies are not as onerous (preregistration, open data), so maybe in that world individual studies would be worth it, but even in that world replication is the only sure fire way to validate results.
  - RandomLensman 3y ago
    Not being replicated isn't the same as not being replicable. There might simply be no interest at the time. But that is no the same as the results not being right or even valuable at some point. Replication is not a be all and end all, because things could replicate even though understanding is wrong (i.e., observation is right in the publication but everything else is wrong). Even the replication itself can be wrong, if certain materials share on unknown contaminant, for example. It addresses some issues, but not all. For some areas it also not clear what replication means. What is it for theology, for example?
    - naasking 3y ago
      > But that is no the same as the results not being right or even valuable at some point. I can't imagine anyone who truly understands the current replication failure rates thinking that a single non-replicated study is valuable for anything other than informing what replications should be attempted. Just think about it: replication rate is generally less than 50%. That means you'll have a better chance of determining the truth on any question posed by a study by flipping a coin than by actually reading the study. > Replication is not a be all and end all, because things could replicate even though understanding is wrong (i.e., observation is right in the publication but everything else is wrong). The observations are the only things that we have to get right. Interpretations can be subject to decades of debate (some QM debates are ongoing a century later), but if the data isn't reliable then you're just wasting time debating falsehoods. Replications are critical to ensuring we have reliable empirical data. > Even the replication itself can be wrong, if certain materials share on unknown contaminant, for example. Indeed, and literally the only way to figure out that such variables exist and are affecting the results are by independent replications. The more replications, the better. One replication is a bare minimum threshold to demonstrate that the process of gathering the data is at least repeatable, in principle. > For some areas it also not clear what replication means. What is it for theology, for example? It makes perfect sense in any context where you're gathering empirical data. For instance, if you're surveying people's interpretation of "free will" [1], then the process via which you probe their views should be replicable. This means being clear about the specific phrasing of the questions asked, the environment in which they were asked, the makeup of the cohort, and so on. [1] https://www.researchgate.net/publication/274892120_Why_Compatibilist_Intuitions_are_not_Mistaken_A_Reply_to_Feltz_and_Millan https://www.researchgate.net/publication/274892120_Why_Compa...