4 ms·

The early days of peer review: five insights from historic reports

61 points by rntn 2y ago

photochemsyn 2y ago
For some reason Nature and Science never want to discuss the other side of peer review - not that involved publication of a paper, but that involved in accepting or rejecting government or other institutional funding proposals. Let's say the NIH announces $1 billion in funding for cancer-related research - surely the system that distributes these funds is worth a little scrutiny, and it also involves peer review.
- deleted 2y ago
  [deleted]
- robotresearcher 2y ago
  Nature and Science regularly discuss these issues. For example, this article from this month: https://www.nature.com/articles/d41586-024-03106-w https://www.nature.com/articles/d41586-024-03106-w
  - photochemsyn 2y ago
    The vast majority of articles on peer review in Nature and Science only refer to journal publication, not funding distribution. Of course they're linked as a publication in Science or Nature helps greatly with the next round of funding applications by all accounts. It matters outside of academics because NIH funding distributions are linked to exclusive patents for pharmaceutical corporations, which represent a massive public subsidy for the pharma sector. This system also accounts for much of the price inflation in the medical drug markets in the USA, and a lack of research into medical uses of out-of-patent (cheap) drugs. See for example: https://medicalxpress.com/news/2023-07-nih-spent-950m-basic-patents.html https://medicalxpress.com/news/2023-07-nih-spent-950m-basic-... > "The article in PLOS ONE titled, "NIH funding for patents that contribute to market exclusivity of drugs approved 2010–2019 and the public interest protections of Bayh-Dole," is the first to look broadly at the NIH funding for the basic and applied research related to new drug approvals and the proportion of this research that leads to patents that give manufacturers market exclusivity."
    - robotresearcher 2y ago
      Sure, it's an important area to discuss in public. I took issue with "never want to discuss", which is not accurate and easily refuted by example. You've backed off to "[not] vast majority of articles on peer review", which is accurate.
swayvil 2y ago
To get primitive... Science translates observation into abstraction. (That's a big leap. And yes, one could make one's own observation. But nobody has the time for that.) But is it a good abstraction? (And then we have all these methods for ensuring that.) Also, why this implicit preference for abstraction? (Yes there's an upside. But consider the downside.)
- asdff 2y ago
  Everything we experience is abstraction
  - swayvil 2y ago
    By abstraction I mean a contrivance of symbols (ie things to which meaning is assigned). Be those symbols ideas, words, patterns of magnetized dots or whatever. Sight, sound, taste, etc are not that. (I suppose even words are not necessarily symbols. I mean, if you don't assign meaning, if you don't interpret it that way, then a word is just a unique pattern of squiggles)
IncreasePosts 2y ago
Intellectuals were writing letters explaining their findings(and asking for feedback) to each other and (math|royal|philosophical) societies all over Europe for 400+ years - shouldn't that count as an earlier form of peer review?
- libraryofbabel 2y ago
  This used to be the view among professional historians of science 25 years ago, but the consensus now in the field is that it’s anachronistic: it back-projects present day views of the scientific process onto the past, akin to looking at the first airplanes and calling then “an earlier form of the Boeing 747.” Depending on who you ask, people will say modern peer review arose in the Cold War or at the earliest in the 19th century. See this encyclopedia entry for a short summary of the issues by a historian: https://ethos.lps.library.cmu.edu/article/id/38/ https://ethos.lps.library.cmu.edu/article/id/38/
  - throw_pm23 2y ago
    Why on earth would someone not call the first airplanes an earlier form of the Boeing 747?
    - throwaway19972 2y ago
      Perhaps because this wouldn't make sense in contexts where you're discussing what distinguishes the Boeing 747 from other planes. If you're comparing to, say, velocipedes, it makes a lot more sense.
    - AnimalMuppet 2y ago
      The first airplanes are an earlier form of the airplane. The 747 is a later form of the airplane. That doesn't make the earlier airplanes an early form of the 747. In the same way, cats are a smaller form of a mammal, and giraffes are a larger form of a mammal. That doesn't make cats a smaller form of a giraffe.
      - 2y ago
YeGoblynQueenne 2y ago
>> Ferlier says that the introduction of the standardized referee questions significantly reduced the amount of time and effort put in by reviewers. “There’s really this understanding in the nineteenth century and very early twentieth century that the peer review is a real discussion,” she says. “After that, it becomes a way of managing the influx of papers for the journal.” Screw that. If you want me to put in the effort to read k pages of your conference or journal paper you better be prepared to read k papers of my review.
- deleted 2y ago
  [deleted]
KingOfCoders 2y ago
When working in research my boss who was very accepted in the field was on many review committees, but I did lots of reviewing (for my failed PhD, "pre-scanning" etc.) - is this the case everywhere? The reviewers by name are not the people doing the actual reviews?
- etrautmann 2y ago
  In neuroscience it was common for PIs to review with the help of one or two trainees. Typically not an outsourcing in my experience but I had good mentors and not everyone operated that way.
- asdff 2y ago
  Its pretty common for the top tier research PIs to rest on their laurels a bit and let their sub PIs or post docs (even grad students) handle the work of writing up the grant proposals and even doing things like peer review on their behalf. These PIs basically exist to support the machine running under their name as they can secure way more grant money than if the lab were broken up.
  - robotresearcher 2y ago
    > Doing things […] on their behalf We train grad students to do reviews by having them do reviews under close supervision, yes. You say it like it’s a bad thing.
    - a_bonobo 2y ago
      Well when I did the reviewing for my PI I rarely got close supervision :) It was more like, 'here's my review', followed by 'I sent it on thanks'
    - asdff 2y ago
      Well it would be one thing if the editor of the journal asked for the grad student to be the reviewer but they asked the PI. Seems a little disingenuous for the PI to pass off that effort as their own. Plus it no doubt takes time away from the grad students other responsibilities.
WhitneyLand 2y ago
There are some little gems in the article, one emphasizes what a badass Turing was. As if it wasn’t enough to essentially invent the field of theoretical computer science and save possibly millions of lives in the war, he then decided in 1952 to casually publish in the field of biology. He had no formal training in biology, and when peer reviewers including the grandson of Charles Darwin commented on his paper “Turing seems to have ignored both sets of comments” lol. Nevertheless the paper turned out to be groundbreaking and is still considered relevant today. Total badass.

61 points by rntn 2y ago

For some reason Nature and Science never want to discuss the other side of peer review - not that involved publication of a paper, but that involved in accepting or rejecting government or other institutional funding proposals. Let's say the NIH announces $1 billion in funding for cancer-related research - surely the system that distributes these funds is worth a little scrutiny, and it also involves peer review.

deleted 2y ago
[deleted]
robotresearcher 2y ago
Nature and Science regularly discuss these issues. For example, this article from this month: https://www.nature.com/articles/d41586-024-03106-w https://www.nature.com/articles/d41586-024-03106-w
- photochemsyn 2y ago
  The vast majority of articles on peer review in Nature and Science only refer to journal publication, not funding distribution. Of course they're linked as a publication in Science or Nature helps greatly with the next round of funding applications by all accounts. It matters outside of academics because NIH funding distributions are linked to exclusive patents for pharmaceutical corporations, which represent a massive public subsidy for the pharma sector. This system also accounts for much of the price inflation in the medical drug markets in the USA, and a lack of research into medical uses of out-of-patent (cheap) drugs. See for example: https://medicalxpress.com/news/2023-07-nih-spent-950m-basic-patents.html https://medicalxpress.com/news/2023-07-nih-spent-950m-basic-... > "The article in PLOS ONE titled, "NIH funding for patents that contribute to market exclusivity of drugs approved 2010–2019 and the public interest protections of Bayh-Dole," is the first to look broadly at the NIH funding for the basic and applied research related to new drug approvals and the proportion of this research that leads to patents that give manufacturers market exclusivity."
  - robotresearcher 2y ago
    Sure, it's an important area to discuss in public. I took issue with "never want to discuss", which is not accurate and easily refuted by example. You've backed off to "[not] vast majority of articles on peer review", which is accurate.

swayvil 2y ago

To get primitive... Science translates observation into abstraction. (That's a big leap. And yes, one could make one's own observation. But nobody has the time for that.) But is it a good abstraction? (And then we have all these methods for ensuring that.) Also, why this implicit preference for abstraction? (Yes there's an upside. But consider the downside.)

asdff 2y ago
Everything we experience is abstraction
- swayvil 2y ago
  By abstraction I mean a contrivance of symbols (ie things to which meaning is assigned). Be those symbols ideas, words, patterns of magnetized dots or whatever. Sight, sound, taste, etc are not that. (I suppose even words are not necessarily symbols. I mean, if you don't assign meaning, if you don't interpret it that way, then a word is just a unique pattern of squiggles)

IncreasePosts 2y ago

Intellectuals were writing letters explaining their findings(and asking for feedback) to each other and (math|royal|philosophical) societies all over Europe for 400+ years - shouldn't that count as an earlier form of peer review?

libraryofbabel 2y ago
This used to be the view among professional historians of science 25 years ago, but the consensus now in the field is that it’s anachronistic: it back-projects present day views of the scientific process onto the past, akin to looking at the first airplanes and calling then “an earlier form of the Boeing 747.” Depending on who you ask, people will say modern peer review arose in the Cold War or at the earliest in the 19th century. See this encyclopedia entry for a short summary of the issues by a historian: https://ethos.lps.library.cmu.edu/article/id/38/ https://ethos.lps.library.cmu.edu/article/id/38/
- throw_pm23 2y ago
  Why on earth would someone not call the first airplanes an earlier form of the Boeing 747?
  - throwaway19972 2y ago
    Perhaps because this wouldn't make sense in contexts where you're discussing what distinguishes the Boeing 747 from other planes. If you're comparing to, say, velocipedes, it makes a lot more sense.
  - AnimalMuppet 2y ago
    The first airplanes are an earlier form of the airplane. The 747 is a later form of the airplane. That doesn't make the earlier airplanes an early form of the 747. In the same way, cats are a smaller form of a mammal, and giraffes are a larger form of a mammal. That doesn't make cats a smaller form of a giraffe.
    - 2y ago

YeGoblynQueenne 2y ago

>> Ferlier says that the introduction of the standardized referee questions significantly reduced the amount of time and effort put in by reviewers. “There’s really this understanding in the nineteenth century and very early twentieth century that the peer review is a real discussion,” she says. “After that, it becomes a way of managing the influx of papers for the journal.” Screw that. If you want me to put in the effort to read k pages of your conference or journal paper you better be prepared to read k papers of my review.

deleted 2y ago
[deleted]

KingOfCoders 2y ago

When working in research my boss who was very accepted in the field was on many review committees, but I did lots of reviewing (for my failed PhD, "pre-scanning" etc.) - is this the case everywhere? The reviewers by name are not the people doing the actual reviews?

etrautmann 2y ago
In neuroscience it was common for PIs to review with the help of one or two trainees. Typically not an outsourcing in my experience but I had good mentors and not everyone operated that way.
asdff 2y ago
Its pretty common for the top tier research PIs to rest on their laurels a bit and let their sub PIs or post docs (even grad students) handle the work of writing up the grant proposals and even doing things like peer review on their behalf. These PIs basically exist to support the machine running under their name as they can secure way more grant money than if the lab were broken up.
- robotresearcher 2y ago
  > Doing things […] on their behalf We train grad students to do reviews by having them do reviews under close supervision, yes. You say it like it’s a bad thing.
  - a_bonobo 2y ago
    Well when I did the reviewing for my PI I rarely got close supervision :) It was more like, 'here's my review', followed by 'I sent it on thanks'
  - asdff 2y ago
    Well it would be one thing if the editor of the journal asked for the grad student to be the reviewer but they asked the PI. Seems a little disingenuous for the PI to pass off that effort as their own. Plus it no doubt takes time away from the grad students other responsibilities.

WhitneyLand 2y ago

There are some little gems in the article, one emphasizes what a badass Turing was. As if it wasn’t enough to essentially invent the field of theoretical computer science and save possibly millions of lives in the war, he then decided in 1952 to casually publish in the field of biology. He had no formal training in biology, and when peer reviewers including the grandson of Charles Darwin commented on his paper “Turing seems to have ignored both sets of comments” lol. Nevertheless the paper turned out to be groundbreaking and is still considered relevant today. Total badass.