3 ms·

This is an excellent step in the right direction, but will it help reduce the absurd volume of studies and the "publish or perish" paradigm?

anamexis 4y ago
I don't see why it would. It doesn't change the incentives.
xiphias2 4y ago
By the time that problem is solved, GPT-8 will be out, and even GPT-4 can be used to help reduce the number of studies (making a table of the studies and asking questions about it). Reproducibility is getting better, but still a huge problem (we can't forget the billions of dollars of funding that went to Alzheimer for a scam paper)
- mike_hearn 4y ago
  Is reproducibility getting better? The last semi-rigorous treatment of the topic I saw suggested there was almost no change over the past decade, at least in the social sciences. It did however use a proxy for reproducibility.
  - xiphias2 4y ago
    Social sciences are inherently political: facebook, tiktok, tinder and onlyfans have all the resources and data to help people getting less depressed (for onlyfans it would be just shutting themselves down, as it's hard for me to think that there's any honest real human connection going on there, which makes both sides depressed over long term). Social workers are just trying to reverse some of the damage companies and governments are doing, but I don't think it will help, even I honestly think they are generally good people doing their best. Maybe disrupting the ad model that just started recently can help, but I believe social studies have no chance to fight against these things.
mike_hearn 4y ago
Fixing that would require a fundamental change in academic incentives. But if you fixed that, it wouldn't be academia anymore. After all, how else do you judge their output? Tangible results that solve a problem for non-researchers? That's what companies pay researchers for. To a large extent academia justifies its own existence by the incentives it gives to researchers.
- ghc 4y ago
  > But if you fixed that, it wouldn't be academia anymore. That's a bold claim worthy of consideration. To assess it, we need to first ask: have the academic incentives always been "publish or perish"? And the answer is...apparently not, according to “Scientific Autonomy, Public Accountability, and the Rise of “Peer Review” in the Cold War United States”. This paper details how the current paradigm is a product of political anti-science-funding sentiment in the 1970s. That tracks with anecdotes from Einstein, Higgs, and Fuller regarding distaste with the modern peer review system. Unfortunately peer review looks like it's actually a policy issue, because it's tied to government funding, so the change in incentives would have to start there. I don't have any answers but at the same time I think it's worth considering the fact that "publish or perish" is not the natural state of academia, but rather a recent phenomenon that appears to coincide with the reproducibility crisis and a decline in the number of large scientific breakthroughs.
  - mike_hearn 4y ago
    If it were the case that peer review was driven by governments, they'd pay people to do it. What's your proposed alternative? No expectations of output at all? Academia has changed many times over the centuries, it's not even clear it has a natural state. As something essentially willed into existence by governments and mega-donors, it's not even really describable as natural. The reproducibility crisis probably isn't new. Maybe public awareness of it is, though I'd say the start of the "crisis" is coming up to its 15th birthday now. But there were people talking about that sort of problem in psychology way back in the 60s and 70s, and examples of bad research claims in other fields that became widely can be found readily when looked for. It seems like reproducibility and other problems are the natural consequence of flooding universities with research grants. Removing the incentive to publish isn't a fix, because then you'd just be flooding universities with money that'd then just disappear, or more likely, get diverted into political campaigning. The money has to be buying something. Because it doesn't buy actual verified results but only papers asserting such results exist, reproducibility problems are an inevitable consequence.

anamexis 4y ago

I don't see why it would. It doesn't change the incentives.

xiphias2 4y ago

By the time that problem is solved, GPT-8 will be out, and even GPT-4 can be used to help reduce the number of studies (making a table of the studies and asking questions about it). Reproducibility is getting better, but still a huge problem (we can't forget the billions of dollars of funding that went to Alzheimer for a scam paper)

mike_hearn 4y ago
Is reproducibility getting better? The last semi-rigorous treatment of the topic I saw suggested there was almost no change over the past decade, at least in the social sciences. It did however use a proxy for reproducibility.
- xiphias2 4y ago
  Social sciences are inherently political: facebook, tiktok, tinder and onlyfans have all the resources and data to help people getting less depressed (for onlyfans it would be just shutting themselves down, as it's hard for me to think that there's any honest real human connection going on there, which makes both sides depressed over long term). Social workers are just trying to reverse some of the damage companies and governments are doing, but I don't think it will help, even I honestly think they are generally good people doing their best. Maybe disrupting the ad model that just started recently can help, but I believe social studies have no chance to fight against these things.

mike_hearn 4y ago

Fixing that would require a fundamental change in academic incentives. But if you fixed that, it wouldn't be academia anymore. After all, how else do you judge their output? Tangible results that solve a problem for non-researchers? That's what companies pay researchers for. To a large extent academia justifies its own existence by the incentives it gives to researchers.

ghc 4y ago
> But if you fixed that, it wouldn't be academia anymore. That's a bold claim worthy of consideration. To assess it, we need to first ask: have the academic incentives always been "publish or perish"? And the answer is...apparently not, according to “Scientific Autonomy, Public Accountability, and the Rise of “Peer Review” in the Cold War United States”. This paper details how the current paradigm is a product of political anti-science-funding sentiment in the 1970s. That tracks with anecdotes from Einstein, Higgs, and Fuller regarding distaste with the modern peer review system. Unfortunately peer review looks like it's actually a policy issue, because it's tied to government funding, so the change in incentives would have to start there. I don't have any answers but at the same time I think it's worth considering the fact that "publish or perish" is not the natural state of academia, but rather a recent phenomenon that appears to coincide with the reproducibility crisis and a decline in the number of large scientific breakthroughs.
- mike_hearn 4y ago
  If it were the case that peer review was driven by governments, they'd pay people to do it. What's your proposed alternative? No expectations of output at all? Academia has changed many times over the centuries, it's not even clear it has a natural state. As something essentially willed into existence by governments and mega-donors, it's not even really describable as natural. The reproducibility crisis probably isn't new. Maybe public awareness of it is, though I'd say the start of the "crisis" is coming up to its 15th birthday now. But there were people talking about that sort of problem in psychology way back in the 60s and 70s, and examples of bad research claims in other fields that became widely can be found readily when looked for. It seems like reproducibility and other problems are the natural consequence of flooding universities with research grants. Removing the incentive to publish isn't a fix, because then you'd just be flooding universities with money that'd then just disappear, or more likely, get diverted into political campaigning. The money has to be buying something. Because it doesn't buy actual verified results but only papers asserting such results exist, reproducibility problems are an inevitable consequence.