6 ms·

Many destructive behaviours are rewarded in a system that is putting emphasis on the quantity and proposed impact of publications. Many scientific fields suffer

Many destructive behaviours are rewarded in a system that is putting emphasis on the quantity and proposed impact of publications. Many scientific fields suffer from behaviours of individuals who are more interested in social status (or just economic security) than in seeking truth. Just like in overall society the humble truth seekers necessary for slowly chiseling out the firm ground of solid knowledge are in the minority. They are definitely disadvantaged in this system, and more likely to be driven out than those who lie, rush or cut corners. At the modern complexity and importance of science, we do need to put much more emphasis on checking whether it is done right. The original vision of the Royal Society involved rigorous peer review, to the point of demanding scientists to demonstrate their experiments in front of their peers: https://www.sciencemuseum.org.uk/objects-and-stories/17th-century-society-transformed-science https://www.sciencemuseum.org.uk/objects-and-stories/17th-ce... The curator of these literal peer reviews held the first paid position in science. Journals do have the profits to hire or otherwise pay peer reviewers full time. They would become experts in their field in the process, and thus more suited for the task than the ones in the current system, where even undergrads are asked to review for high-profile outlets like NeurIPS. Also, full-time rigorous peer reviewing would be an interesting career prospect for many current scientists. And here is your startup idea..

jampekka 3y ago
Rigorous peer review, let alone paid (which causes its own issues beyond money), is totally infeasible with the current publication pressure. It's hard to find reviewers even with the current lax criteria of both reviewers and quality of their review (the latter is more problematic). The publication volume is just way way way too high. But researchers who don't publish multiple articles per year, regardless if you have anything of value to publish, perish. If you don't churn out paper-per-year in your PhD, you don't get the PhD. Much of the manuscripts that get submitted to journals are incredibly bad. Most are just bad. But as both editor and reviewer, I usually let them be published out of pity if the stuff isn't blatantly wrong; the poor PhD student's whole career is on the line. Journals being full of crap is not SO bad within science because everybody knows they are full of crap. But if an "outsider" thinks that being published in a peer-reviewed journal, even a "good" journal, means that the article isn't crap, it can be literally life-or-death (like in this case). Academia is broken because it's being run as a business whose purpose is to churn out papers. Welcome to neoliberalism.
- cscheid 3y ago
  Although the above comment might sound negative and harsh, it is a perfect distillation of modern research-oriented academic environments. (I was a moderately-successful professor in these environments. I woke up one day and simply couldn't do it anymore.)
- stanford_labrat 3y ago
  Publication rate DURING your PhD is highly variable from field to field...I don't know what field you are in but generally speaking across my 3 fields students are required to have at least 1 from the entire PhD to graduate. Some even have none published by time of graduation. 1 per year is a ridiculous standard.
  - jampekka 3y ago
    This does vary across countries, and fields. In UK I was a bit surprised that you're not expected to publish at all before post-doc. In Finland usually at least 3 peer reviewed (first author) papers are required for a PhD in my field (cognitive science). In some fields (e.g. many engineering fields) even more. And PhD grants are typically for three to four years. It is ridiculous and the paper quality is what you'd expect.
  - ska 3y ago
    It is wildly different, true. In some areas, it is not uncommon to see PhD "thesis" that is essentially an intro stapled to 3-4 papers. It's also true that publication rates post PhD vary wildly as well, e.g. expectations for a tenure packet. However, it's also fair to say that expected publication rate has significantly grown universally. A generation or two ago, a solid career could be built on a handful of high impact papers. That's hard to imagine now.
    - deleted 3y ago
      [deleted]
    - jampekka 3y ago
      That thesis format is the norm in Finland in sciency fields and also in some arts. You can write a monograph (essentially a book) instead but that's frowned upon. Especially by admin because the papers bring the univerisity more money than monographs. Nowadays piles of inconsequential papers where you contributed little but your name is almost a requirement for a solid career. It's horrible.
swatcoder 3y ago
It's way way bigger than that. The whole world changed since those early days of modern science. During those days, the science you were talking about was one of countless hobby clubs for aristocratic circles, where bored rich folk could dedicate themselves to one upping each other in the discovery game their peers and recent predecessors had invented. There were certainly cheaters then, too, but the whole thing was very insular and the only people who even cared about it were the people who were in on the game themselves. So excelling within the rules and elaborate demonstrations of the game were all part of the club sport. 400 years later, billions of people across five or six generations have been told that this game is the secret to human prosperity on earth and that the more its played, the more aligned we'll be with the truth of the universe, the longer we'll all live, the more leisure and luxury we'll all enjoy, etc That's a whole different game! The demand for scientific output has gone from an exhibition sport shared among a small and snobby circle to a replacement for an eroded theology. Trillion dollar governments and globe-spanning trillion dollar industries now (ostensibly) make their billion dollar decisions based on each day's summary of the sport, and billions of people dreaming about prosperity or salvation await the next big game's result as though it were a demonstration of divine grace. There are now so many games being played, by so many people, with so many bets and spectators, that incentives are unfathomable and referees are sparse and cheating is rampant. Unfortunately, you're not going to clean all that up by telling some journals to reshuffle their revenue allocations.
- wredue 3y ago
  Sorry but who actually thinks that being a scientist is the “key to prosperity”? I think far and away most scientists become researchers despite the fact that it’s generally not at all a prosperous venture.
  - swatcoder 3y ago
    Not "being a scientist": "science". Science is seen as the road to human (not just personal) propsperity by people educated in the last 100-ish years, which is mostly everybody now, which puts extreme pressure to perform/produce on what was once a pure little hobby sport whose spectators were almost all invested as players themselves.
hibikir 3y ago
It's not necessarily even status, just feeding themselves: Publish or die, and you aren't publishing if your conclusions aren't positive and strong. In medicine it's especially tough, as testing is among the most expensive and the number of available data points is quite low, so one gets to fignt over individual data points that make a difference. Did this person not follow the protocol, and I have a paper, or not? I've seen something similar when working in agriculture: Early tests of new plant strains are low data, because a company will start by testing so many experimental plants that it'd be unaffordable to test them all very strongly. This makes people really argue about single data points in those early tests. Was this chunk of the field contaminated? Attacked by a wild animal? But there at least the interest in fraud is small. If a breeder gets a stinker through, it just goes into trials on the other hemisphere, where it's planted in an order of magnitude more fields, and therefore just lead to disappointment 6 months later. With medicine, the cost of replication is so high, and often takes so many years, that it's not just that an honest mistake is catastrophic, but that the difference in outcomes for the person doing the curation is so high, one doesn't have to be all that dishonest to make biased decisions that will lead to a strong career.
- BurningFrog 3y ago
  > It's not necessarily even status, just feeding themselves: Publish or die, and you aren't publishing if your conclusions aren't positive and strong. The wider science needs to learn a lesson from the Economics science: Incentives Matter. A lot.
  - jampekka 3y ago
    Much of the problem is that wider science has taken too many lessons from economics. Incentives matter a lot but creating incentives that work is almost impossible. Science used to run on ethics (like many other professions). But we learned from economists that there are no such thing, only utility (money) maximizers.
    - nvm0n2 3y ago
      Incentives that work are easy. Do research in corporate labs. That's all it takes. Now there is an incentive to do work that replicates (because it is intended to be used to build useful things), there are people responsible for detecting and resolving fraud (managers), they are incentivized to do so by a mix of carrots and sticks some of them legal. Fact is, corporate R&D doesn't have this relentless problem with reproducibility. It's academic output that does, because academics only care about getting a paper published and don't expect that anyone will use their results. Often they don't even make their code or data available at all because it's not to their advantage for others to be able to replicate their work, as that would yield fewer papers. But this is all wrong. Science exists to be used in technology, not for its own sake.
mbreese 3y ago
> Also, full-time rigorous peer reviewing would be an interesting career prospect for many current scientists. And here is your startup idea.. I don’t know about this. (But I don’t have any answers for the question either). If you’re a full time peer reviewer, first, are you really a peer? But more importantly, your motivations change. No longer are you looking to try to see if the paper is worthy of publishing or if it is sound; instead, your motivation is to push through as many papers as possible. When getting paid depends on approving papers, the quality will drop. Maybe the problem is where the money exchange occurs. What about if authors paid to have their paper reviewed, instead of published? Currently, journals only get paid when a paper is published. What about if they got paid to review the paper at all? It would limit the paper submissions to Nature/Science/Cell, but you’d be paying for a high quality review (which often makes a paper better). You might even have luck with decoupling reviewers from journals completely… make the journals compete over the best (already) reviewed papers.
- bumby 3y ago
  >instead, your motivation is to push through as many papers as possible. I think this is an assumption that doesn't have to hold true in practice. Maybe it's biased by the 'publish or perish' paradigm that's pervaded academia, but there's no reason to replicate the same problem elsewhere.
deleted 3y ago
[deleted]

jampekka 3y ago

Rigorous peer review, let alone paid (which causes its own issues beyond money), is totally infeasible with the current publication pressure. It's hard to find reviewers even with the current lax criteria of both reviewers and quality of their review (the latter is more problematic). The publication volume is just way way way too high. But researchers who don't publish multiple articles per year, regardless if you have anything of value to publish, perish. If you don't churn out paper-per-year in your PhD, you don't get the PhD. Much of the manuscripts that get submitted to journals are incredibly bad. Most are just bad. But as both editor and reviewer, I usually let them be published out of pity if the stuff isn't blatantly wrong; the poor PhD student's whole career is on the line. Journals being full of crap is not SO bad within science because everybody knows they are full of crap. But if an "outsider" thinks that being published in a peer-reviewed journal, even a "good" journal, means that the article isn't crap, it can be literally life-or-death (like in this case). Academia is broken because it's being run as a business whose purpose is to churn out papers. Welcome to neoliberalism.

cscheid 3y ago
Although the above comment might sound negative and harsh, it is a perfect distillation of modern research-oriented academic environments. (I was a moderately-successful professor in these environments. I woke up one day and simply couldn't do it anymore.)
stanford_labrat 3y ago
Publication rate DURING your PhD is highly variable from field to field...I don't know what field you are in but generally speaking across my 3 fields students are required to have at least 1 from the entire PhD to graduate. Some even have none published by time of graduation. 1 per year is a ridiculous standard.
- jampekka 3y ago
  This does vary across countries, and fields. In UK I was a bit surprised that you're not expected to publish at all before post-doc. In Finland usually at least 3 peer reviewed (first author) papers are required for a PhD in my field (cognitive science). In some fields (e.g. many engineering fields) even more. And PhD grants are typically for three to four years. It is ridiculous and the paper quality is what you'd expect.
- ska 3y ago
  It is wildly different, true. In some areas, it is not uncommon to see PhD "thesis" that is essentially an intro stapled to 3-4 papers. It's also true that publication rates post PhD vary wildly as well, e.g. expectations for a tenure packet. However, it's also fair to say that expected publication rate has significantly grown universally. A generation or two ago, a solid career could be built on a handful of high impact papers. That's hard to imagine now.
  - deleted 3y ago
    [deleted]
  - jampekka 3y ago
    That thesis format is the norm in Finland in sciency fields and also in some arts. You can write a monograph (essentially a book) instead but that's frowned upon. Especially by admin because the papers bring the univerisity more money than monographs. Nowadays piles of inconsequential papers where you contributed little but your name is almost a requirement for a solid career. It's horrible.

swatcoder 3y ago

It's way way bigger than that. The whole world changed since those early days of modern science. During those days, the science you were talking about was one of countless hobby clubs for aristocratic circles, where bored rich folk could dedicate themselves to one upping each other in the discovery game their peers and recent predecessors had invented. There were certainly cheaters then, too, but the whole thing was very insular and the only people who even cared about it were the people who were in on the game themselves. So excelling within the rules and elaborate demonstrations of the game were all part of the club sport. 400 years later, billions of people across five or six generations have been told that this game is the secret to human prosperity on earth and that the more its played, the more aligned we'll be with the truth of the universe, the longer we'll all live, the more leisure and luxury we'll all enjoy, etc That's a whole different game! The demand for scientific output has gone from an exhibition sport shared among a small and snobby circle to a replacement for an eroded theology. Trillion dollar governments and globe-spanning trillion dollar industries now (ostensibly) make their billion dollar decisions based on each day's summary of the sport, and billions of people dreaming about prosperity or salvation await the next big game's result as though it were a demonstration of divine grace. There are now so many games being played, by so many people, with so many bets and spectators, that incentives are unfathomable and referees are sparse and cheating is rampant. Unfortunately, you're not going to clean all that up by telling some journals to reshuffle their revenue allocations.

wredue 3y ago
Sorry but who actually thinks that being a scientist is the “key to prosperity”? I think far and away most scientists become researchers despite the fact that it’s generally not at all a prosperous venture.
- swatcoder 3y ago
  Not "being a scientist": "science". Science is seen as the road to human (not just personal) propsperity by people educated in the last 100-ish years, which is mostly everybody now, which puts extreme pressure to perform/produce on what was once a pure little hobby sport whose spectators were almost all invested as players themselves.

hibikir 3y ago

It's not necessarily even status, just feeding themselves: Publish or die, and you aren't publishing if your conclusions aren't positive and strong. In medicine it's especially tough, as testing is among the most expensive and the number of available data points is quite low, so one gets to fignt over individual data points that make a difference. Did this person not follow the protocol, and I have a paper, or not? I've seen something similar when working in agriculture: Early tests of new plant strains are low data, because a company will start by testing so many experimental plants that it'd be unaffordable to test them all very strongly. This makes people really argue about single data points in those early tests. Was this chunk of the field contaminated? Attacked by a wild animal? But there at least the interest in fraud is small. If a breeder gets a stinker through, it just goes into trials on the other hemisphere, where it's planted in an order of magnitude more fields, and therefore just lead to disappointment 6 months later. With medicine, the cost of replication is so high, and often takes so many years, that it's not just that an honest mistake is catastrophic, but that the difference in outcomes for the person doing the curation is so high, one doesn't have to be all that dishonest to make biased decisions that will lead to a strong career.

BurningFrog 3y ago
> It's not necessarily even status, just feeding themselves: Publish or die, and you aren't publishing if your conclusions aren't positive and strong. The wider science needs to learn a lesson from the Economics science: Incentives Matter. A lot.
- jampekka 3y ago
  Much of the problem is that wider science has taken too many lessons from economics. Incentives matter a lot but creating incentives that work is almost impossible. Science used to run on ethics (like many other professions). But we learned from economists that there are no such thing, only utility (money) maximizers.
  - nvm0n2 3y ago
    Incentives that work are easy. Do research in corporate labs. That's all it takes. Now there is an incentive to do work that replicates (because it is intended to be used to build useful things), there are people responsible for detecting and resolving fraud (managers), they are incentivized to do so by a mix of carrots and sticks some of them legal. Fact is, corporate R&D doesn't have this relentless problem with reproducibility. It's academic output that does, because academics only care about getting a paper published and don't expect that anyone will use their results. Often they don't even make their code or data available at all because it's not to their advantage for others to be able to replicate their work, as that would yield fewer papers. But this is all wrong. Science exists to be used in technology, not for its own sake.

mbreese 3y ago

> Also, full-time rigorous peer reviewing would be an interesting career prospect for many current scientists. And here is your startup idea.. I don’t know about this. (But I don’t have any answers for the question either). If you’re a full time peer reviewer, first, are you really a peer? But more importantly, your motivations change. No longer are you looking to try to see if the paper is worthy of publishing or if it is sound; instead, your motivation is to push through as many papers as possible. When getting paid depends on approving papers, the quality will drop. Maybe the problem is where the money exchange occurs. What about if authors paid to have their paper reviewed, instead of published? Currently, journals only get paid when a paper is published. What about if they got paid to review the paper at all? It would limit the paper submissions to Nature/Science/Cell, but you’d be paying for a high quality review (which often makes a paper better). You might even have luck with decoupling reviewers from journals completely… make the journals compete over the best (already) reviewed papers.

bumby 3y ago
>instead, your motivation is to push through as many papers as possible. I think this is an assumption that doesn't have to hold true in practice. Maybe it's biased by the 'publish or perish' paradigm that's pervaded academia, but there's no reason to replicate the same problem elsewhere.

deleted 3y ago

[deleted]