4 ms·

Whether the "peer review" experiment has succeeded or failed is debatable... but anecdotally, everyone I know in academia feels the current publish-or-perish pe

by cs702 4y ago

Whether the "peer review" experiment has succeeded or failed is debatable... but anecdotally, everyone I know in academia feels the current publish-or-perish peer-reviewed system is BROKEN: It incentivizes everyone to pursue low-risk, incremental, mostly inconsequential, easy-to-explain research with high certainty of short-term success, instead of high-risk, exploratory, consequential, hard-to-explain research with high uncertainty of long-term payoff. This is true: > From antiquity to modernity, scientists wrote letters and circulated monographs, and the main barriers stopping them from communicating their findings were the cost of paper, postage, or a printing press, or on rare occasions, the cost of a visit from the Catholic Church. Scientific journals appeared in the 1600s, but they operated more like magazines or newsletters, and their processes of picking articles ranged from “we print whatever we get” to “the editor asks his friend what he thinks” to “the whole society votes.” Sometimes journals couldn’t get enough papers to publish, so editors had to go around begging their friends to submit manuscripts, or fill the space themselves. Scientific publishing remained a hodgepodge for centuries. This is true as well: > That all changed after World War II. Governments poured funding into research, and they convened “peer reviewers” to ensure they weren’t wasting their money on foolish proposals. That funding turned into a deluge of papers, and journals that previously struggled to fill their pages now struggled to pick which articles to print. Reviewing papers before publication, which was “quite rare” until the 1960s, became much more common. Then it became universal. Finally, this is also true: > Now pretty much every journal uses outside experts to vet papers, and papers that don’t please reviewers get rejected. You can still write to your friends about your findings, but hiring committees and grant agencies act as if the only science that exists is the stuff published in peer-reviewed journals. This is the grand experiment we’ve been running for six decades.

inglor_cz 4y ago
"everyone I know in academia feels the current publish-or-perish peer-reviewed system is BROKEN: It incentivizes everyone to pursue low-risk, incremental, mostly inconsequential, easy-to-explain research with high certainty of short-term success, instead of high-risk, exploratory, consequential, hard-to-explain research with high uncertainty of long-term payoff." That is what I keep hearing from my academic acquaintances as well - When a measure becomes a target, it ceases to be a good measure. And this is what happened with publish or perish. Let's not forget that Katalin Karikó, a giant in the mRNA field, was demoted by UPenn in 1997 because they considered her work unproductive. But mRNA was precisely the topic that required a multi-decade commitment. Fortunately she was stubborn like hell. God know how many other people give up after being degraded like that.
- watwut 4y ago
  That is complain against the grant system and the way funds are distributed. It is not complain abour peer review itself.
  - djha-skin 4y ago
    The article makes it clear that peer review and funding are intricately linked and that one would not exist without the other, at least in the beginning.
    - watwut 4y ago
      It does not make clear that at all. It does not claim peer review can not exist without things those scientista complain about (publish or perish etc).
- xipho 4y ago
  > But mRNA was precisely the topic that required a multi-decade commitment. It all comes down to this ^ for me. Advances take decades from start to finish. Institutions/organziations (including funders) need to start not just funding innovation (1-5 year cycles), but investing, really committing, to 20-100 year plans, at minimum. My biggest fear is that hands are thrown up in the air, and science turns into yet another "content" producer (there are signs, to me, that this is exactly where we are headed). Can't produce science^D content fast enough? Skip peer-review. Community of experts getting in your way? Ignore them. We need to get science into tik-tok, because think of the reach for STEM! Really, do we need science to equate to 15-second dopamine hits? None of this to say that there aren't problems with peer review, but, at least in the field I'm in, peer review is critical, useful, and an important aspect of becoming a scientist. If anything, learning to peer-review teaches one how to critically evaluate others work. Are there people who aren't taught how to Peer review? Sure. Does this mean that we should stop peer-review? The logic doesn't follow.
- lazyjeff 4y ago
  I've had my ups and downs with peer review. I'm trying to collect stories of papers from ideation to publication, while also describing the struggle for my own papers. I've written the backstories for 30 of my papers here: https://jeffhuang.com/struggle_for_each_paper/ https://jeffhuang.com/struggle_for_each_paper/ Looking at my past experiences for all my papers, I can only conclude that the process is more nuanced than can be described in a tagline like "peer review is broken" or "publishing is random", so I'm interested in hearing everyone else's experiences too.
biomcgary 4y ago
The Sydney Brenner quote around the mid-point of the page captures the reality quite well, "it’s simply a regression to the mean". Or, given the typical peer review mechanisms, regression below the mean. If the original author is above average and the paper has three reviewers, any one of which can block publication, what is the likelihood that the paper will be blocked by someone less competent than the author? If reviewers are sampled at random from the scientific population (they aren't, it's usually worse) then blocking the paper is almost guaranteed to be blocked by someone less competent than the author. A few journals are slowly moving toward a better system where publication happens first, then the reviews are published. As a scientist, I would certainly prefer to read the reviews without them blocking/delaying publication. However, the real problem with peer review is its role in awarding grants, not in publishing papers. I have no idea how to fix that problem without introducing other problems. Too much incentive to game the system.
- sandgiant 4y ago
  This argument makes no sense. Why would "less than average" referees block a paper? And why would editors select referees from the population at random? I don't know about other fields, but that's certainly not how things work in physics. I would say it's more like a club where everyone knows each other. If you are authoring a paper you often know who the potential referees are (those that published most in the field), and you politely ask the editor to exclude referees that are in open conflict with you. Editors usually abide and you get a reasonable review of your paper. I agree about most other issues raised about peer review in general and related incentives, but the issues raised here are not some that I recognize.
  - biomcgary 4y ago
    In biology (my area), it is quite frequent for reviewers to "suggest" additional analyses/experiments that reference the the reviewers work or sub-field, which delays publication and inflates citations of reviewers. Thus, some people seem to make a strategy of volunteering for review to get more citations. My understanding is that physics has used arXiv much longer and more consistently than biology has used bioRxiv, so communication suffers less from the impact of delayed (not just blocked) publication. Physics is a smaller research community than biology, so it is not surprising that club effects have more pronounced. Some less funded sub-fields of biology definitely have that feel (e.g., ecology and evolution), but anything remotely biomedical seems to suffer from the rando reviewer effect much more. In my experience, the historic "squishiness" of biology and the rapid growth of NIH funding (and hence graduate students mills) leads to a larger population of less rigorous thinkers than in physics. We don't have math requirements to filter those folks out.

by cs702 4y ago

inglor_cz 4y ago

"everyone I know in academia feels the current publish-or-perish peer-reviewed system is BROKEN: It incentivizes everyone to pursue low-risk, incremental, mostly inconsequential, easy-to-explain research with high certainty of short-term success, instead of high-risk, exploratory, consequential, hard-to-explain research with high uncertainty of long-term payoff." That is what I keep hearing from my academic acquaintances as well - When a measure becomes a target, it ceases to be a good measure. And this is what happened with publish or perish. Let's not forget that Katalin Karikó, a giant in the mRNA field, was demoted by UPenn in 1997 because they considered her work unproductive. But mRNA was precisely the topic that required a multi-decade commitment. Fortunately she was stubborn like hell. God know how many other people give up after being degraded like that.

watwut 4y ago
That is complain against the grant system and the way funds are distributed. It is not complain abour peer review itself.
- djha-skin 4y ago
  The article makes it clear that peer review and funding are intricately linked and that one would not exist without the other, at least in the beginning.
  - watwut 4y ago
    It does not make clear that at all. It does not claim peer review can not exist without things those scientista complain about (publish or perish etc).
xipho 4y ago
> But mRNA was precisely the topic that required a multi-decade commitment. It all comes down to this ^ for me. Advances take decades from start to finish. Institutions/organziations (including funders) need to start not just funding innovation (1-5 year cycles), but investing, really committing, to 20-100 year plans, at minimum. My biggest fear is that hands are thrown up in the air, and science turns into yet another "content" producer (there are signs, to me, that this is exactly where we are headed). Can't produce science^D content fast enough? Skip peer-review. Community of experts getting in your way? Ignore them. We need to get science into tik-tok, because think of the reach for STEM! Really, do we need science to equate to 15-second dopamine hits? None of this to say that there aren't problems with peer review, but, at least in the field I'm in, peer review is critical, useful, and an important aspect of becoming a scientist. If anything, learning to peer-review teaches one how to critically evaluate others work. Are there people who aren't taught how to Peer review? Sure. Does this mean that we should stop peer-review? The logic doesn't follow.
lazyjeff 4y ago
I've had my ups and downs with peer review. I'm trying to collect stories of papers from ideation to publication, while also describing the struggle for my own papers. I've written the backstories for 30 of my papers here: https://jeffhuang.com/struggle_for_each_paper/ https://jeffhuang.com/struggle_for_each_paper/ Looking at my past experiences for all my papers, I can only conclude that the process is more nuanced than can be described in a tagline like "peer review is broken" or "publishing is random", so I'm interested in hearing everyone else's experiences too.

biomcgary 4y ago

The Sydney Brenner quote around the mid-point of the page captures the reality quite well, "it’s simply a regression to the mean". Or, given the typical peer review mechanisms, regression below the mean. If the original author is above average and the paper has three reviewers, any one of which can block publication, what is the likelihood that the paper will be blocked by someone less competent than the author? If reviewers are sampled at random from the scientific population (they aren't, it's usually worse) then blocking the paper is almost guaranteed to be blocked by someone less competent than the author. A few journals are slowly moving toward a better system where publication happens first, then the reviews are published. As a scientist, I would certainly prefer to read the reviews without them blocking/delaying publication. However, the real problem with peer review is its role in awarding grants, not in publishing papers. I have no idea how to fix that problem without introducing other problems. Too much incentive to game the system.

sandgiant 4y ago
This argument makes no sense. Why would "less than average" referees block a paper? And why would editors select referees from the population at random? I don't know about other fields, but that's certainly not how things work in physics. I would say it's more like a club where everyone knows each other. If you are authoring a paper you often know who the potential referees are (those that published most in the field), and you politely ask the editor to exclude referees that are in open conflict with you. Editors usually abide and you get a reasonable review of your paper. I agree about most other issues raised about peer review in general and related incentives, but the issues raised here are not some that I recognize.
- biomcgary 4y ago
  In biology (my area), it is quite frequent for reviewers to "suggest" additional analyses/experiments that reference the the reviewers work or sub-field, which delays publication and inflates citations of reviewers. Thus, some people seem to make a strategy of volunteering for review to get more citations. My understanding is that physics has used arXiv much longer and more consistently than biology has used bioRxiv, so communication suffers less from the impact of delayed (not just blocked) publication. Physics is a smaller research community than biology, so it is not surprising that club effects have more pronounced. Some less funded sub-fields of biology definitely have that feel (e.g., ecology and evolution), but anything remotely biomedical seems to suffer from the rando reviewer effect much more. In my experience, the historic "squishiness" of biology and the rapid growth of NIH funding (and hence graduate students mills) leads to a larger population of less rigorous thinkers than in physics. We don't have math requirements to filter those folks out.