4 ms·

Journals should not shed out tiny payments for peer reviews by overworked faculty, they should hire teams of reviewers with diverse specialisations full time. B

Journals should not shed out tiny payments for peer reviews by overworked faculty, they should hire teams of reviewers with diverse specialisations full time. By decent quality assurance on the massive amount of papers each year publishers could actually do something valuable for science.

michaelmior 6y ago
I don't necessarily disagree with paying reviewers, but I think there is some value to having people who are actively working on problems in the same area being the ones doing the reviewing. Yes, it's additional work that we aren't paid for, but personally I enjoy it. I do agree though that for some it can create undue pressure so there may be something in the middle there that works well.
- DanielleMolloy 6y ago
  I understand this point, yet to correctly assess whether a typical paper in my field - for instance - can be trusted you would need a modern neural networks researcher, a statistician, a cognitive scientist, a neuroscientist and perhaps a physicist. All would have to know about common fallacies in their respective fields and should optimally discuss with another. Faculty reviewers are not only overworked, they have accumulated biases, can be overconfident about their expertise and in the worst case have malicious incentive to prohibit publication of your results. I’m certain that a team of specialists who review as their full time job would do a better job than at least half of the reviewers I encountered (especially those at high-profile AI conferences). I enjoy reviewing too and am doing my best to be constructive, but I’m not sufficiently qualified for most of the offers I receive.
  - michaelmior 6y ago
    I suppose a lot of this is dependent on the field. I think anyone who has had their work peer reviewed has many stories of reviews from people who seem to have completely misunderstood the work in question. Certainly there can be accumulated biases and overconfidence among faculty reviewers, but I don't see how that would be less prevalent among paid reviewers.
  - rocqua 6y ago
    Would a person / team with the qualifications you mention best spend his time on reviewing such work, or should they also be doing some research. My point is, the only way to get the necessary skillset for reviewing is to actually do research. Hence it is peer-review. Once you are doing scientific research, there generally isn't anyone left who is 'better at it then you' so the best you can do is have it be reviewed by people who are 'roughly as good at it as you'. To ensure the long-term accuracy, we need both attempts at reproduction, and critical reading by people post publication. Both of those should have the option to cause the article to be retracted.
piokoch 6y ago
That would be difficult in a fields with a vast range of specializations (physics for sure). They would really need an army of reviewers to cover everything from astronomy to particle physics.
sokoloff 6y ago
Why would society want to encourage people with the deepest specialty knowledge to work for journals and away from the bench?
- DanielleMolloy 6y ago
  Lots of these highly qualified researches are forced to leave science right now, the reasons all too often not being their qualification. There would be a large pool of good researchers to draw from for this.
  - sokoloff 6y ago
    Can each individual journal (say Nature Neuroscience, Neuron, EMBO, Acta, Lancet, Trends, Annals of Neurology, PLoS, and Current Biology) each feed the dozens of neuroscientist specialists who would be needed as employees to do these reviews? How would each of those specialists retain currency in the field by just doing reviews? Isn't that what the editor panels do more effectively today (of course farming out the detailed/specialist aspects to actual peer reviewers)?
    - DanielleMolloy 6y ago
      Hiring: Are you aware of the discussion around Elseviers profit margins? Various fields are considering abandoning journals (or have abandoned them long ago, see cryptography) since it is increasingly difficult to identify the value of a for-profit publisher for science. They will need to redefine their business model if sufficient scientists stop playing along for free. „Currency“: Broad expertise, knowledge about methods and ideas are gathered by reading literature in-depth, which most scientists have little time for in modern science (hence the reluctance to engage in a long peer review process). Working as a paid reviewer for a few years can lead to a scientist who is informed very well about the state of the broader field and its various fallacies, and could move on from there. We have very few of those. Editors: We wouldn’t have various discussions around peer review if the current system would be working satisfactorily well and nothing needs to change.

michaelmior 6y ago

I don't necessarily disagree with paying reviewers, but I think there is some value to having people who are actively working on problems in the same area being the ones doing the reviewing. Yes, it's additional work that we aren't paid for, but personally I enjoy it. I do agree though that for some it can create undue pressure so there may be something in the middle there that works well.

DanielleMolloy 6y ago
I understand this point, yet to correctly assess whether a typical paper in my field - for instance - can be trusted you would need a modern neural networks researcher, a statistician, a cognitive scientist, a neuroscientist and perhaps a physicist. All would have to know about common fallacies in their respective fields and should optimally discuss with another. Faculty reviewers are not only overworked, they have accumulated biases, can be overconfident about their expertise and in the worst case have malicious incentive to prohibit publication of your results. I’m certain that a team of specialists who review as their full time job would do a better job than at least half of the reviewers I encountered (especially those at high-profile AI conferences). I enjoy reviewing too and am doing my best to be constructive, but I’m not sufficiently qualified for most of the offers I receive.
- michaelmior 6y ago
  I suppose a lot of this is dependent on the field. I think anyone who has had their work peer reviewed has many stories of reviews from people who seem to have completely misunderstood the work in question. Certainly there can be accumulated biases and overconfidence among faculty reviewers, but I don't see how that would be less prevalent among paid reviewers.
- rocqua 6y ago
  Would a person / team with the qualifications you mention best spend his time on reviewing such work, or should they also be doing some research. My point is, the only way to get the necessary skillset for reviewing is to actually do research. Hence it is peer-review. Once you are doing scientific research, there generally isn't anyone left who is 'better at it then you' so the best you can do is have it be reviewed by people who are 'roughly as good at it as you'. To ensure the long-term accuracy, we need both attempts at reproduction, and critical reading by people post publication. Both of those should have the option to cause the article to be retracted.

piokoch 6y ago

That would be difficult in a fields with a vast range of specializations (physics for sure). They would really need an army of reviewers to cover everything from astronomy to particle physics.

sokoloff 6y ago

Why would society want to encourage people with the deepest specialty knowledge to work for journals and away from the bench?

DanielleMolloy 6y ago
Lots of these highly qualified researches are forced to leave science right now, the reasons all too often not being their qualification. There would be a large pool of good researchers to draw from for this.
- sokoloff 6y ago
  Can each individual journal (say Nature Neuroscience, Neuron, EMBO, Acta, Lancet, Trends, Annals of Neurology, PLoS, and Current Biology) each feed the dozens of neuroscientist specialists who would be needed as employees to do these reviews? How would each of those specialists retain currency in the field by just doing reviews? Isn't that what the editor panels do more effectively today (of course farming out the detailed/specialist aspects to actual peer reviewers)?
  - DanielleMolloy 6y ago
    Hiring: Are you aware of the discussion around Elseviers profit margins? Various fields are considering abandoning journals (or have abandoned them long ago, see cryptography) since it is increasingly difficult to identify the value of a for-profit publisher for science. They will need to redefine their business model if sufficient scientists stop playing along for free. „Currency“: Broad expertise, knowledge about methods and ideas are gathered by reading literature in-depth, which most scientists have little time for in modern science (hence the reluctance to engage in a long peer review process). Working as a paid reviewer for a few years can lead to a scientist who is informed very well about the state of the broader field and its various fallacies, and could move on from there. We have very few of those. Editors: We wouldn’t have various discussions around peer review if the current system would be working satisfactorily well and nothing needs to change.