5 ms·

Publishers trial paying peer reviewers – what did they find?

31 points by xqcgrek2 2y ago

westurner 2y ago
> USD $250 How much deep research does $250 yield by comparison? Knowledge market > Examples; Google Answers, Yahoo Answers, https://en.wikipedia.org/wiki/Knowledge_market#Examples https://en.wikipedia.org/wiki/Knowledge_market#Examples
- pjdesno 2y ago
  I'm not sure why one would compare reviews by acknowledged experts in a field with stuff written by anonymous randos, and it seems highly unlikely that anyone with the appropriate qualifications would be lurking on some mechanical turk-like site. I'm also deeply suspicious of the confidentiality of anything sent to one of those sites. However this does suggest the idea that a high-powered university in a low-income country might be able to cut a deal to provide reviewing services...
- moomin 2y ago
  It’ll get you an electrician for about three hours in London. How long do these papers take to read critically?
  - voxl 2y ago
    One full work day to due decently. Two full work days to do well.
- tdeck 2y ago
  You can get 50 reviews on Fiverr for that price!
withinboredom 2y ago
There's this guy I usually have on in the background on youtube who replicates chemistry experiments -- or attempts to. It's pretty rare to see him find a paper that doesn't exaggerate yields or go into enough details, and he has to guess things.
- datadrivenangel 2y ago
  You don't exaggerate yields, you just publish the best one you get out of a dozen attempts. Chemistry is messy.
  - thyristan 2y ago
    That, in science, is called "lying". Either you publish the range of results, the average plus standard deviation or average plus standard deviation of a subset with the exclusion criteria and exclusion range. Picking a result is a lie, plain and simple, and messiness is not an excuse.
    - mattmanser 2y ago
      See my comment too, you jump to lying, but as the GP said, chemistry is messy.
      - thyristan 2y ago
        Any other science is messy as well. Truck passing by on the nearby road? Oops, my physics experiment got shaken, results look messy. Lab animal caught a cold? Oops, genetics experiment now has messy data. Atmosphere is turbulent and some shitty starlink satellite passed by at the wrong moment? Oops, my stellar spectra are messy now. Imperfection in my test ingot? Oops, now my tensile strength measurements have messy data because a few ripped too early... It is the nature of experimental science to deal with messiness. And dealing with it means being honest about it. You write it like it happened, find the problems in the messy parts of your data, exclude that and explain the why and how. Hand-picked results and just omitting data you find inconvenient is not science, its fraud. When I am allowed to just pick one result I can show you a perpetuum mobile, cold fusion, superhuman intelligence in mice and tons of other newsworthy things...
        mattmanser 2y ago
        Can I ask if you've done any actual commercial work in any science? From the way you're talking, I'm going to guess you're an armchair commentator. One person performing an unfamiliar experiment once is going to get lower yields and occasional failures.
jruohonen 2y ago
> Some 53% of researchers accepted the invitation to review when offered payment, compared with 48% of those who received a standard, non-paid offer. On average, paid reviews came in one day earlier than unpaid ones. Does not sound like notable effects to either end. (I was once offered a payment for a peer review, but declined it.)
- mmooss 2y ago
  Don't overlook the other experiment's results.
mmooss 2y ago
What are the requirements of a review? And what is the marketplace for someone meeting those requirements? What expertise is required - someone who researches the same questions? Same general domain? Adjacent domain? And how long does it take? I imagine that depends on many details. Finally, what are they reviewing for? Is it a once-over for errors in method? Something like grading a student paper?
- tsumnia 2y ago
  Speaking as a CS Education reviewer, some of the criteria can be "signing up to review", though solicitation is often sent to professionals in the domain (through personal requests or blanket email campaigns), as well as through respective mailing lists. I review papers for I think 4-5 conferences, mostly because I have colleagues that serve/publish in those spaces (you declare conflicts of interest to avoid bias). Each publisher/conference have their own reviewing guidelines to follow, but at least for the conferences I've reviewed for they include: a summary (2-5 sentences tops), the strengths of, the weaknesses of the research, and potentially your opinion on the piece. You are typically asked to include your familiarity with the research space since you may be reviewing methodologies that you were not explicitly trained in. This all distills into a metric that effectively reflects "this paper should be accepted/not accepted" which is then handed to a 'senior' reviewer to summarize for the conference to decide. All of my conferences are double-blind single submission, but I have colleagues that are able to respond to reviewer critiques. Most conferences recognize things like grammatical issues can happen, so reviewers are asked to only point them out rather than use them as a basis for rejection; however if the paper is riddled with mistakes, then it can be grounds for rejection. Likewise, since CS Education is a combination of CS and cognitive psychology, some of the discussion can be attributed toward "appropriateness for CS education research". For example, I once reviewed a paper that clearly was including theater-based education techniques but had CS shoehorned in one paragraph (that was it). Alternatively, measuring time delays in student responses to a tutoring system can help distinguish when students become distracted or take a break.
  - mmooss 2y ago
    Thanks. Someone told me that the 'blind' review doesn't often work because they already know who is doing what in their field.
drgo 2y ago
I think what publishers need to do is retain reviewers (possibly on part-time basis); many retired scientists can benefit from those opportunities and it is a way to keep senior scientists engaged in their fields. For most submitted papers, there is no need for the reviewer to be sub-specialized in the paper's field (most reviews done by the sub specialists are actually done by their postdocs and grad students) and the hiring process (and subsequent evaluation) is ought to be more effective and speedier than randomly contacting people to beg for reviews. Until the review process is taken more seriously by publishers and journal editors, the quality of published science continues to deteriorate.
odyssey7 2y ago
Peer review is work. The workers are subject to capitalism. Pay them or capitalism will optimize the quality unfavorably

31 points by xqcgrek2 2y ago

> USD $250 How much deep research does $250 yield by comparison? Knowledge market > Examples; Google Answers, Yahoo Answers, https://en.wikipedia.org/wiki/Knowledge_market#Examples https://en.wikipedia.org/wiki/Knowledge_market#Examples

pjdesno 2y ago
I'm not sure why one would compare reviews by acknowledged experts in a field with stuff written by anonymous randos, and it seems highly unlikely that anyone with the appropriate qualifications would be lurking on some mechanical turk-like site. I'm also deeply suspicious of the confidentiality of anything sent to one of those sites. However this does suggest the idea that a high-powered university in a low-income country might be able to cut a deal to provide reviewing services...
moomin 2y ago
It’ll get you an electrician for about three hours in London. How long do these papers take to read critically?
- voxl 2y ago
  One full work day to due decently. Two full work days to do well.
tdeck 2y ago
You can get 50 reviews on Fiverr for that price!

withinboredom 2y ago

There's this guy I usually have on in the background on youtube who replicates chemistry experiments -- or attempts to. It's pretty rare to see him find a paper that doesn't exaggerate yields or go into enough details, and he has to guess things.

datadrivenangel 2y ago
You don't exaggerate yields, you just publish the best one you get out of a dozen attempts. Chemistry is messy.
- thyristan 2y ago
  That, in science, is called "lying". Either you publish the range of results, the average plus standard deviation or average plus standard deviation of a subset with the exclusion criteria and exclusion range. Picking a result is a lie, plain and simple, and messiness is not an excuse.
  - mattmanser 2y ago
    See my comment too, you jump to lying, but as the GP said, chemistry is messy.
    - thyristan 2y ago
      Any other science is messy as well. Truck passing by on the nearby road? Oops, my physics experiment got shaken, results look messy. Lab animal caught a cold? Oops, genetics experiment now has messy data. Atmosphere is turbulent and some shitty starlink satellite passed by at the wrong moment? Oops, my stellar spectra are messy now. Imperfection in my test ingot? Oops, now my tensile strength measurements have messy data because a few ripped too early... It is the nature of experimental science to deal with messiness. And dealing with it means being honest about it. You write it like it happened, find the problems in the messy parts of your data, exclude that and explain the why and how. Hand-picked results and just omitting data you find inconvenient is not science, its fraud. When I am allowed to just pick one result I can show you a perpetuum mobile, cold fusion, superhuman intelligence in mice and tons of other newsworthy things...
      - mattmanser 2y ago
        Can I ask if you've done any actual commercial work in any science? From the way you're talking, I'm going to guess you're an armchair commentator. One person performing an unfamiliar experiment once is going to get lower yields and occasional failures.

jruohonen 2y ago

> Some 53% of researchers accepted the invitation to review when offered payment, compared with 48% of those who received a standard, non-paid offer. On average, paid reviews came in one day earlier than unpaid ones. Does not sound like notable effects to either end. (I was once offered a payment for a peer review, but declined it.)

mmooss 2y ago
Don't overlook the other experiment's results.

mmooss 2y ago

What are the requirements of a review? And what is the marketplace for someone meeting those requirements? What expertise is required - someone who researches the same questions? Same general domain? Adjacent domain? And how long does it take? I imagine that depends on many details. Finally, what are they reviewing for? Is it a once-over for errors in method? Something like grading a student paper?

tsumnia 2y ago
Speaking as a CS Education reviewer, some of the criteria can be "signing up to review", though solicitation is often sent to professionals in the domain (through personal requests or blanket email campaigns), as well as through respective mailing lists. I review papers for I think 4-5 conferences, mostly because I have colleagues that serve/publish in those spaces (you declare conflicts of interest to avoid bias). Each publisher/conference have their own reviewing guidelines to follow, but at least for the conferences I've reviewed for they include: a summary (2-5 sentences tops), the strengths of, the weaknesses of the research, and potentially your opinion on the piece. You are typically asked to include your familiarity with the research space since you may be reviewing methodologies that you were not explicitly trained in. This all distills into a metric that effectively reflects "this paper should be accepted/not accepted" which is then handed to a 'senior' reviewer to summarize for the conference to decide. All of my conferences are double-blind single submission, but I have colleagues that are able to respond to reviewer critiques. Most conferences recognize things like grammatical issues can happen, so reviewers are asked to only point them out rather than use them as a basis for rejection; however if the paper is riddled with mistakes, then it can be grounds for rejection. Likewise, since CS Education is a combination of CS and cognitive psychology, some of the discussion can be attributed toward "appropriateness for CS education research". For example, I once reviewed a paper that clearly was including theater-based education techniques but had CS shoehorned in one paragraph (that was it). Alternatively, measuring time delays in student responses to a tutoring system can help distinguish when students become distracted or take a break.
- mmooss 2y ago
  Thanks. Someone told me that the 'blind' review doesn't often work because they already know who is doing what in their field.

drgo 2y ago

I think what publishers need to do is retain reviewers (possibly on part-time basis); many retired scientists can benefit from those opportunities and it is a way to keep senior scientists engaged in their fields. For most submitted papers, there is no need for the reviewer to be sub-specialized in the paper's field (most reviews done by the sub specialists are actually done by their postdocs and grad students) and the hiring process (and subsequent evaluation) is ought to be more effective and speedier than randomly contacting people to beg for reviews. Until the review process is taken more seriously by publishers and journal editors, the quality of published science continues to deteriorate.

odyssey7 2y ago

Peer review is work. The workers are subject to capitalism. Pay them or capitalism will optimize the quality unfavorably