6 ms·

It seems remiss to leave out the publish or perish dynamic that requires anyone pursuing a career in academia to churn out publications with little to no regard

by daoboy 3y ago

It seems remiss to leave out the publish or perish dynamic that requires anyone pursuing a career in academia to churn out publications with little to no regard for their quality. Especially in light of the evidence in recent years that peer review is a poor filter for fraud. Deincentivising an endless stream of trash publications will both cut down on the BS and narrow the field of research one might try to reproduce. I don't have strong feelings about the optimal number of papers that should be published, but my sense is that we are running an order of magnitude above that number.

edgyquant 3y ago
Why not just require independent reproduction to even publish?
- mikrl 3y ago
  That would also incentivize the paper writers to be exacting in describing their methodology and lead to higher quality papers. If a paper needed to clear the bars of - can be reproduced by reading the paper and/or a few meetings with the original team - data is independently reproducible to within the usual statistical parameters It would presumably increase the signal:noise of research.
  - BurningFrog 3y ago
    I like this. Requiring reproduction is several bridges to far. But requiring reproducibility seems complete reasonable!
    - swayvil 3y ago
      So you just take their word for it? Pretend that reading about it is equivalent to seeing it?
      - anonymouskimmer 3y ago
        Many scientists reading an article will seek to extend the work in the article, not duplicate it. If the extension fails then they may spend the effort to replicate the original results. Even reproduction doesn't solve the problem of theoretical errors: https://news.ycombinator.com/item?id=36230450 https://news.ycombinator.com/item?id=36230450
        swayvil 3y ago
        Reproduction strikes me as the first significant step in extending and debugging. But that's just me.
        anonymouskimmer 3y ago
        Engineering and Science go hand in hand, but one is one and the other is the other.
      - BurningFrog 3y ago
        That's the current system. I suggest an improvement.
    - notyofriend 3y ago
      If it’s not worth reproducing it’s not worth publishing imo
      - anonymouskimmer 3y ago
        Until grants explicitly grant money for an independent lab to reproduce the research they are paying for (effectively almost doubling the grant amount), no one is going to spend their hard-won funds on reproducing someone else's research. At least not in a timely manner. What such a requirement would do is dramatically scale back research, and highly incentivize researchers to lie, or the "independent labs" to fudge in order to keep drawing funding from their collaborators.
        vlovich123 3y ago
        Is it a jobs program or are we investing money to get to the truth of things?
        anonymouskimmer 3y ago
        Granting agencies, whether government or NGO, typically want certain kinds of applied science results. I think "truth" is more assumed than the explicit goal.
        Qem 3y ago
        > Until grants explicitly grant money for an independent lab to reproduce the research they are paying for (effectively almost doubling the grant amount) I think a lot of money would be saved by avoiding researchers to waste resources in dead ends, by trying to build upon results that can't be reproduced. So not exactly double.
        anonymouskimmer 3y ago
        I completely agree, but unfortunately it's not the same money that is in the initial grant, it's someone else's money. So by granting more money granting agencies can theoretically save some other granting agency's money, but not necessarily their own. Meanwhile that other granting agency will get a lot more bang for their buck (and quantitatively more prestigious results) than the first granting agency. Bad incentive structures.
  - medstrom 3y ago
    Nitpick but "a few meetings with the original team" is what we have today, ya?
- bluGill 3y ago
  Most science isn't easily reproduced. Often it takes a lot of time and money to get the right equipment. The worst case is super colliders where there is one in the world capable of doing it and so we have to trust the staff. Even simple cases though is still significant effort. There isn't much low hanging you could reproduce it tonight in your basement science left.
  - Ilnsk 3y ago
    Yeah ok but most science doesn't need a supercollider. Most of the erroneous science is in the social sciences, psych especially, and there really isn't any equivalent of a supercollider in psychology. The only possible equivalent is a massive study of like 200+ subjects but once you're at that scale you can be pretty confident that your statistics have converged, anyway. The real issue is low sample size(<100 although most studies done are <30) psychology studies which are by and large relatively easy to reproduce.
    - vlovich123 3y ago
      I don’t think all the problems that cause lack of reproducibility come from small sample sizes. For example, P-hacking is a thing (intentionally or not) and larger sample sizes don’t solve that. Experiment registration can help so you can track negative results but that doesn’t help if there’s no reproduction attempt (ie you could just have gotten lucky). There’s also straight up fraud you have to deal with. The point is, op is right that it’s expensive. The computer industry that we’re in claims to be data driven but I’ve observed numerous poor quality studies being done to drive decisions that I’m pretty jaded (no reproduction, poor sample sizes, skewed sample sizes where it’s employees, etc etc). And these are smart people where the decisions being made can impact the financial outcome.
  - YetAnotherNick 3y ago
    The funny thing is that supercollider people try way harder to reproduce it whenever possible than social or nutritional sciences group. e.g. Higgs boson detection was only made public when it was found by two independent teams looking at different data(CMS and ATLAS). Also astronomy is good example where reproduction is the norm.
- danaris 3y ago
  This would also require a reduction in the "publish or perish" pressure, because it would significantly lengthen the time from study design to publishing. Furthermore, who's going to be doing that independent reproduction? How are you going to incentivize that? If you require independent reproduction for every paper in a field where that's even possible, that's going to mean either you need to double the rate of running experiments in that field (unlikely), or double the number of researchers in that field (also unlikely) if you want to maintain anywhere close to the current rate of papers being published. (And again, even if you could double the number of researchers, you still need to provide enough of an incentive to reproduce that literally everyone is doing so for half of the studies they run. And you'd somehow have to ensure that you don't end up with fun, interesting studies getting 15 different groups reproducing them, while less-flashy studies that are still important fundamental science languish for years without anyone caring to try.) Given how much of "publish or perish" is institutional inertia and culture, the idea that it could be systemically "nudged" to allow for this kind of shift is essentially a fantasy. Far too many of the Old Guard who are already in charge of the tenure committees would say, "Well, I had to publish 327 papers to be given tenure, and kids these days have so many newfangled things that let them write papers faster; why shouldn't we require them to publish 327 papers, too??"
  - notahacker 3y ago
    To add to your "who's going to be doing that independent reproduction?" point, the whole purpose of the journals is to communicate to the rest of the scientific community that there's an interesting result here that might want further examination. Sure, in the absence of the ability to disseminate your findings via the usual channel, you might be able to instruct your grad students to reproduce or beg one of your regular collaborators to reproduce, but then the independence of the experiments is dubious. And otherwise, academics aren't exactly searching for unpublished papers to replicate so the original can be published.
  - anonymouskimmer 3y ago
    > If you require independent reproduction for every paper in a field where that's even possible, that's going to mean either you need to double the rate of running experiments in that field (unlikely), or double the number of researchers in that field (also unlikely) You don't have to quite double the number of researchers or experiments. It typically takes less work to replicate final results than all of the work getting to the point of good results. Presumably a lot of the replication would be outsourced to CROs. Though since the CROs would be focused on replication of results alone, any theoretical errors or errors in experiment design wouldn't be discovered. Replicating results will mostly catch fabricators. It won't catch nearly all of the far more important to the replication crisis bad experimental design.
- RandomLensman 3y ago
  Is it even clear that it would be worth it? Or do you propose a drastic reduction in papers to go with it, i.e., only things valuable enough to have someone replicate it are published?
  - civilitty 3y ago
    If research is not worth replicating, then how could it be anything but a useless contribution meant only to bolster someone’s publish or perish career? It just seems that the entire scientific publishing industry is there to support a jobs and prestige program. Any science is just a side effect that somehow justifies the whole racket, from NSF budget to postdoc dinner table.
    - RandomLensman 3y ago
      Not strictly true: it only needs to matter not enough at the time. But, yes, arguably a fair amount of publications are largely irrelevant beyond a few people. Societies fund many things for many reasons, not sure science is worth singling out here.
      - medstrom 3y ago
        I don't mind the funding, but inclusion in a journal dresses it up with the blessing of a Scientific Result if the news ever cite it. We could have a separate kind of journal for not-yet-reproduced results, and somehow ensure that the prestige is zero (or equivalent to just posting the study on a blog).
        RandomLensman 3y ago
        We could, but I doubt it would stop the news from picking stuff up. They often go with press releases anyway, not a journal citation. You would need incentives on the news side then to stop that.
    - wredue 3y ago
      Reproducibility in some fields are difficult/impossible for a lot of reasons (funding, candidates, etc). Suppressing publishing at the same time as actively trashing the ability to even study seems like a recipe for disaster way above publishing bad papers to me.
- mnky9800n 3y ago
  Whose going to pay for that?

by daoboy 3y ago

edgyquant 3y ago

Why not just require independent reproduction to even publish?

mikrl 3y ago
That would also incentivize the paper writers to be exacting in describing their methodology and lead to higher quality papers. If a paper needed to clear the bars of - can be reproduced by reading the paper and/or a few meetings with the original team - data is independently reproducible to within the usual statistical parameters It would presumably increase the signal:noise of research.
- BurningFrog 3y ago
  I like this. Requiring reproduction is several bridges to far. But requiring reproducibility seems complete reasonable!
  - swayvil 3y ago
    So you just take their word for it? Pretend that reading about it is equivalent to seeing it?
    - anonymouskimmer 3y ago
      Many scientists reading an article will seek to extend the work in the article, not duplicate it. If the extension fails then they may spend the effort to replicate the original results. Even reproduction doesn't solve the problem of theoretical errors: https://news.ycombinator.com/item?id=36230450 https://news.ycombinator.com/item?id=36230450
      - swayvil 3y ago
        Reproduction strikes me as the first significant step in extending and debugging. But that's just me.
        anonymouskimmer 3y ago
        Engineering and Science go hand in hand, but one is one and the other is the other.
    - BurningFrog 3y ago
      That's the current system. I suggest an improvement.
  - notyofriend 3y ago
    If it’s not worth reproducing it’s not worth publishing imo
    - anonymouskimmer 3y ago
      Until grants explicitly grant money for an independent lab to reproduce the research they are paying for (effectively almost doubling the grant amount), no one is going to spend their hard-won funds on reproducing someone else's research. At least not in a timely manner. What such a requirement would do is dramatically scale back research, and highly incentivize researchers to lie, or the "independent labs" to fudge in order to keep drawing funding from their collaborators.
      - vlovich123 3y ago
        Is it a jobs program or are we investing money to get to the truth of things?
        anonymouskimmer 3y ago
        Granting agencies, whether government or NGO, typically want certain kinds of applied science results. I think "truth" is more assumed than the explicit goal.
      - Qem 3y ago
        > Until grants explicitly grant money for an independent lab to reproduce the research they are paying for (effectively almost doubling the grant amount) I think a lot of money would be saved by avoiding researchers to waste resources in dead ends, by trying to build upon results that can't be reproduced. So not exactly double.
        anonymouskimmer 3y ago
        I completely agree, but unfortunately it's not the same money that is in the initial grant, it's someone else's money. So by granting more money granting agencies can theoretically save some other granting agency's money, but not necessarily their own. Meanwhile that other granting agency will get a lot more bang for their buck (and quantitatively more prestigious results) than the first granting agency. Bad incentive structures.
- medstrom 3y ago
  Nitpick but "a few meetings with the original team" is what we have today, ya?
bluGill 3y ago
Most science isn't easily reproduced. Often it takes a lot of time and money to get the right equipment. The worst case is super colliders where there is one in the world capable of doing it and so we have to trust the staff. Even simple cases though is still significant effort. There isn't much low hanging you could reproduce it tonight in your basement science left.
- Ilnsk 3y ago
  Yeah ok but most science doesn't need a supercollider. Most of the erroneous science is in the social sciences, psych especially, and there really isn't any equivalent of a supercollider in psychology. The only possible equivalent is a massive study of like 200+ subjects but once you're at that scale you can be pretty confident that your statistics have converged, anyway. The real issue is low sample size(<100 although most studies done are <30) psychology studies which are by and large relatively easy to reproduce.
  - vlovich123 3y ago
    I don’t think all the problems that cause lack of reproducibility come from small sample sizes. For example, P-hacking is a thing (intentionally or not) and larger sample sizes don’t solve that. Experiment registration can help so you can track negative results but that doesn’t help if there’s no reproduction attempt (ie you could just have gotten lucky). There’s also straight up fraud you have to deal with. The point is, op is right that it’s expensive. The computer industry that we’re in claims to be data driven but I’ve observed numerous poor quality studies being done to drive decisions that I’m pretty jaded (no reproduction, poor sample sizes, skewed sample sizes where it’s employees, etc etc). And these are smart people where the decisions being made can impact the financial outcome.
- YetAnotherNick 3y ago
  The funny thing is that supercollider people try way harder to reproduce it whenever possible than social or nutritional sciences group. e.g. Higgs boson detection was only made public when it was found by two independent teams looking at different data(CMS and ATLAS). Also astronomy is good example where reproduction is the norm.
danaris 3y ago
This would also require a reduction in the "publish or perish" pressure, because it would significantly lengthen the time from study design to publishing. Furthermore, who's going to be doing that independent reproduction? How are you going to incentivize that? If you require independent reproduction for every paper in a field where that's even possible, that's going to mean either you need to double the rate of running experiments in that field (unlikely), or double the number of researchers in that field (also unlikely) if you want to maintain anywhere close to the current rate of papers being published. (And again, even if you could double the number of researchers, you still need to provide enough of an incentive to reproduce that literally everyone is doing so for half of the studies they run. And you'd somehow have to ensure that you don't end up with fun, interesting studies getting 15 different groups reproducing them, while less-flashy studies that are still important fundamental science languish for years without anyone caring to try.) Given how much of "publish or perish" is institutional inertia and culture, the idea that it could be systemically "nudged" to allow for this kind of shift is essentially a fantasy. Far too many of the Old Guard who are already in charge of the tenure committees would say, "Well, I had to publish 327 papers to be given tenure, and kids these days have so many newfangled things that let them write papers faster; why shouldn't we require them to publish 327 papers, too??"
- notahacker 3y ago
  To add to your "who's going to be doing that independent reproduction?" point, the whole purpose of the journals is to communicate to the rest of the scientific community that there's an interesting result here that might want further examination. Sure, in the absence of the ability to disseminate your findings via the usual channel, you might be able to instruct your grad students to reproduce or beg one of your regular collaborators to reproduce, but then the independence of the experiments is dubious. And otherwise, academics aren't exactly searching for unpublished papers to replicate so the original can be published.
- anonymouskimmer 3y ago
  > If you require independent reproduction for every paper in a field where that's even possible, that's going to mean either you need to double the rate of running experiments in that field (unlikely), or double the number of researchers in that field (also unlikely) You don't have to quite double the number of researchers or experiments. It typically takes less work to replicate final results than all of the work getting to the point of good results. Presumably a lot of the replication would be outsourced to CROs. Though since the CROs would be focused on replication of results alone, any theoretical errors or errors in experiment design wouldn't be discovered. Replicating results will mostly catch fabricators. It won't catch nearly all of the far more important to the replication crisis bad experimental design.
RandomLensman 3y ago
Is it even clear that it would be worth it? Or do you propose a drastic reduction in papers to go with it, i.e., only things valuable enough to have someone replicate it are published?
- civilitty 3y ago
  If research is not worth replicating, then how could it be anything but a useless contribution meant only to bolster someone’s publish or perish career? It just seems that the entire scientific publishing industry is there to support a jobs and prestige program. Any science is just a side effect that somehow justifies the whole racket, from NSF budget to postdoc dinner table.
  - RandomLensman 3y ago
    Not strictly true: it only needs to matter not enough at the time. But, yes, arguably a fair amount of publications are largely irrelevant beyond a few people. Societies fund many things for many reasons, not sure science is worth singling out here.
    - medstrom 3y ago
      I don't mind the funding, but inclusion in a journal dresses it up with the blessing of a Scientific Result if the news ever cite it. We could have a separate kind of journal for not-yet-reproduced results, and somehow ensure that the prestige is zero (or equivalent to just posting the study on a blog).
      - RandomLensman 3y ago
        We could, but I doubt it would stop the news from picking stuff up. They often go with press releases anyway, not a journal citation. You would need incentives on the news side then to stop that.
  - wredue 3y ago
    Reproducibility in some fields are difficult/impossible for a lot of reasons (funding, candidates, etc). Suppressing publishing at the same time as actively trashing the ability to even study seems like a recipe for disaster way above publishing bad papers to me.
mnky9800n 3y ago
Whose going to pay for that?