8 ms·

One of my profs once remarked, "All of science is done on a volunteer basis." He was talking about peer review, which--as crucial as it is--is not something yo

by rhelz 3y ago

One of my profs once remarked, "All of science is done on a volunteer basis." He was talking about peer review, which--as crucial as it is--is not something you get paid for. Writing a review--a good review--is 1) hard work, 2) can only be done by somebody who has spent years in postgraduate study, and 3) takes up a lot of time, which has many other demands on it. The solution? Its obvious. In a free market, how do you signal if you want more of something to be produced? Bueller? Bueller? Yeah, that's right, you gotta pay for it. This cost should just be estimated and factored into the original grant proposals--if its not worth $5k or $10k to fund a round of peer review, and perhaps also funds to run confirming experiments--well, then its probably not research worth doing in the first place. So yeah, write up the grants to include the actual full cost of doing and publishing the research. It would be a great way for starving grad students to earn some coin, and the experience gained in running confirming experiments would be invaluable to help them get that R.A. position or postdoc.

bumby 3y ago
I don't disagree with the proposed idea of paying for review, but I would prefer also to have guardrails to ensure a good review. I would be willing to pay for a good review because it makes the paper/investigation better. But let's face it: under the current paradigm, there are also a lot of really bad reviews. It's one thing when it's apparent that a reviewer doesn't understand something because of a lack of clarity in the writing. But it's also extremely frustrating when it's obvious the reviewer hasn't even bothered to carefully read the manuscript. Under a payment paradigm, we need mechanisms to limit the incentive to maximize throughput as a means of getting the most pay and instead maximize the review quality. I assume there'd be good ways to do that, but I don't know what those would be.
- heikkilevanto 3y ago
  So, we just need a meta-review to review the reviews. At a cost, of course. And in order to keep that honest, we need a meta-meta-review...
  - notatoad 3y ago
    we need twitter community notes for science
  - currymj 3y ago
    many CS conferences have something literally called a "meta-review" and then there are further senior people who read and oversee the meta-reviews. it stops there though.
    - abhgh 3y ago
      Unfortunately the state of meta-review is similar to that of reviews. Rarely delves deeper, mostly acts as a summarizer for the independent reviews.
    - MichaelZuo 3y ago
      Who picks the meta reviewers?
  - bumby 3y ago
    Or possibly just a way to review the reviewers. This opens itself up to competitor bias, though, so it would need to be thought out in a way to minimize that.
  - rhelz 3y ago
    chuckle recall, a paper is given to 3 or 4 reviewers. No need for hierarchies of reviewers; it’s more like a jury; if all the reviewers more or less come to the same conclusion we can have a high confidence that the decision is the correct one. Under the proposed plan, if one of the reviewers gave a review which was radically different, or otherwise obviously slap-dash job, payment could be withheld and another reviewer commissioned.
nitwit005 3y ago
Unfortunately, what you'll actually incentivize is spending as little effort as possible to get the money paid out.
- matwood 3y ago
  Opposed to now where it appears lots of science is peer reviewed with all the problems found later?
  - nitwit005 3y ago
    Replacing a broken system, with a broken and also expensive system, does not sound like an improvement.
  - andyferris 3y ago
    It might cost $100k - $1m (or more) to repeat the work and run replications. The $5k - $10k mentioned earlier would be enough to allow time reading and thinking and checking some stuff on pen-and-paper.
    - michaelt 3y ago
      > The $5k - $10k mentioned earlier would be enough to allow time reading and thinking and checking some stuff on pen-and-paper. The average postdoc in the US earns $32.81/hour, according to the results of some googling. Even taking overheads into account, $5k should cover more than a week's full time work.
      - sseagull 3y ago
        In what area of science would it take only a week or two to replicate? It might take several days to a week of literature review just to fully understand the problem. Then you might need equipment, chemicals, cultures, etc. Then depending on the area of science, doing the actual experiment could take several weeks (waiting for reactions, computer simulations, etc). Then possibly tricky analysis and statistics on top of that. Nowadays, science is deep
eviks 3y ago
The outcome would most likely be exactly this: "it's probably research not worth doing in first place" (and why would you want to signal you want more busy work?)
geysersam 3y ago
Or, research just gets published online free of charge for everyone to access, and important work will prove itself over time by being discovered, discussed and by becoming influential. If anyone wants to change something about an article (the writing, the structure of the paper, or anything else a reviewer might want to edit) they can just do it and publish a new version. If people like the new version better, good, if they don't they can read the old version. Peer review as a filter for publishing is terrible in a time when making a few megabytes of text and images accessible is literally free. If anyone wants to run a content aggregator (aka a Journal) they just do it. If they want to change something about the article before it's approved for the aggregator they can contact the authors or ask someone to review it or whatever. Just make it accessible.
- rhelz 3y ago
  Whether or not your papers pass peer review—-and which journals it is published in—are important criteria for hiring, tenure, whether your grants are funded, etc. If you get rid of peer review, it’s not science. It’s just a vanity press.
  - bumby 3y ago
    >If you get rid of peer review, it’s not science. It’s just a vanity press. If you believe replicability is central to science, the current paradigm doesn't necessarily converge on science either. And when people are graded on how many publications they garner, it borders on turning publication into a symbol of status rather than one of science.
    - rhelz 3y ago
      > If you believe replicability is central to science. I do believe that, but it doesn’t matter what anybody believes, replicatable experiments and results, which peers can review and agree on, are the soul of science. Without that it’s not science, it’s just creative writing.
      - 3y ago
PaulHoule 3y ago
And also has limited value as we practice it today. A useful review would involve: (a) "This paper won't be accepted by Cochrane for meta analysis", "N=20 get out of here", ... (b) Researchers provided their data files and Jupyter notebooks, the reviewer got them to run (c) Reviewers attempt their own analysis for at least some of the data (think of the model of accounting where auditors look at a sample of the books) (d) Reviewers come visit the lab and take a look at the apparatus (e) Something like a PhD defense (f) Summarize/formalize discussion of a paper that happens at a conference or online venue (g) 5 months, 5 years, or 40 years later (once in my case) somebody goes through the math line-by-line and finds a mistake on equation 178 that propagates and affects the rest of the calculation. This knowledge ought to be captured and "stapled" to the paper. I wouldn't say peer review is useless, I think it did improve papers I wrote a little but reviewers do not put enough effort into to reliably catch validity problems. If you believe in meta-analysis, which you should, read this book https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1114087/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1114087/ the first one (a) is really important because the overwhelming majority of papers are excluded in Cochrane reviews for most medical topics. If 80-90% of papers in some fields are not suitable for inference that raises tough questions such as "Why does work like this get published?" and "Why does this work get funded?" If we got half as many papers but 80-90% of them were usable that would be a revolution.
- rhelz 3y ago
  > If you believe in meta-analysis, which you should, I view meta-analysis as being like those mortgage-backed securities which crashed the world back in 2008. I mean, yeah, theoretically, a bond whose yield is a weighted average of mortgages should be less risky than any one of the mortgages in it. But....the devil is in the details. When I start seeing meta-analysis which claim that, e.g., masks don't protect you from a respiratory illness which spreads by coughing and sneezing, or that vaccines are at best worthless and at worse cause autism, wellll..... ....I have to conclude that garbage-in, garbage-out.
- rhelz 3y ago
  > I wouldn't say peer review is useless, I think it did improve papers I wrote a little Let's get back to basics here. What is the whole point of doing things scientifically in the first place? I.e. what is so special about the combination of 1) communicable descriptions of experiments, which can be re-run by anybody, and 2) the performing of said experiments and reporting the results, and 3) having others review and reproduce the results? Why are those 3 practices soooo special? Its because they are designed to either a) persuade everybody to agreement with you that your results are valid, or b) persuade you that you are wrong. I.e.. the scientific method is uniquely designed for the purpose of compelling agreement among peers. Its the 2nd best method we have of reaching consensus (the first best being the methods of proof used in mathematics. ) This is important because even if you make a new gee-whiz discovery, its useless unless other people agree that it is real. And unlike politics, or a beauty contest, or an aristocracy, it doesn't matter who you are, if your results are observable and reproducible, other scientists will agree with you, and your discovery has some chance of being generally useful. And that's why they are anonymous--its supposed to cut through the biases and prejudices which, alas, sometimes prevent us from coming to consensus. The point of peer review is to get a read on whether or not the author of the paper has justified their claims in enough detail that the results are likely to compile agreement. Yeah, we see if 3 or 4 people can agree on it, before asking everybody to agree on it. Whoda thunk? Point being, peer review isn't to help you improve your paper. > reviewers do not put enough effort into to reliably catch validity problems. Yes. This. Any why don't they? Because there's only so much work you can expect somebody to do for free. Lets pay them.

by rhelz 3y ago

bumby 3y ago

I don't disagree with the proposed idea of paying for review, but I would prefer also to have guardrails to ensure a good review. I would be willing to pay for a good review because it makes the paper/investigation better. But let's face it: under the current paradigm, there are also a lot of really bad reviews. It's one thing when it's apparent that a reviewer doesn't understand something because of a lack of clarity in the writing. But it's also extremely frustrating when it's obvious the reviewer hasn't even bothered to carefully read the manuscript. Under a payment paradigm, we need mechanisms to limit the incentive to maximize throughput as a means of getting the most pay and instead maximize the review quality. I assume there'd be good ways to do that, but I don't know what those would be.

heikkilevanto 3y ago
So, we just need a meta-review to review the reviews. At a cost, of course. And in order to keep that honest, we need a meta-meta-review...
- notatoad 3y ago
  we need twitter community notes for science
- currymj 3y ago
  many CS conferences have something literally called a "meta-review" and then there are further senior people who read and oversee the meta-reviews. it stops there though.
  - abhgh 3y ago
    Unfortunately the state of meta-review is similar to that of reviews. Rarely delves deeper, mostly acts as a summarizer for the independent reviews.
  - MichaelZuo 3y ago
    Who picks the meta reviewers?
- bumby 3y ago
  Or possibly just a way to review the reviewers. This opens itself up to competitor bias, though, so it would need to be thought out in a way to minimize that.
- rhelz 3y ago
  chuckle recall, a paper is given to 3 or 4 reviewers. No need for hierarchies of reviewers; it’s more like a jury; if all the reviewers more or less come to the same conclusion we can have a high confidence that the decision is the correct one. Under the proposed plan, if one of the reviewers gave a review which was radically different, or otherwise obviously slap-dash job, payment could be withheld and another reviewer commissioned.

nitwit005 3y ago

Unfortunately, what you'll actually incentivize is spending as little effort as possible to get the money paid out.

matwood 3y ago
Opposed to now where it appears lots of science is peer reviewed with all the problems found later?
- nitwit005 3y ago
  Replacing a broken system, with a broken and also expensive system, does not sound like an improvement.
- andyferris 3y ago
  It might cost $100k - $1m (or more) to repeat the work and run replications. The $5k - $10k mentioned earlier would be enough to allow time reading and thinking and checking some stuff on pen-and-paper.
  - michaelt 3y ago
    > The $5k - $10k mentioned earlier would be enough to allow time reading and thinking and checking some stuff on pen-and-paper. The average postdoc in the US earns $32.81/hour, according to the results of some googling. Even taking overheads into account, $5k should cover more than a week's full time work.
    - sseagull 3y ago
      In what area of science would it take only a week or two to replicate? It might take several days to a week of literature review just to fully understand the problem. Then you might need equipment, chemicals, cultures, etc. Then depending on the area of science, doing the actual experiment could take several weeks (waiting for reactions, computer simulations, etc). Then possibly tricky analysis and statistics on top of that. Nowadays, science is deep

eviks 3y ago

The outcome would most likely be exactly this: "it's probably research not worth doing in first place" (and why would you want to signal you want more busy work?)

geysersam 3y ago

Or, research just gets published online free of charge for everyone to access, and important work will prove itself over time by being discovered, discussed and by becoming influential. If anyone wants to change something about an article (the writing, the structure of the paper, or anything else a reviewer might want to edit) they can just do it and publish a new version. If people like the new version better, good, if they don't they can read the old version. Peer review as a filter for publishing is terrible in a time when making a few megabytes of text and images accessible is literally free. If anyone wants to run a content aggregator (aka a Journal) they just do it. If they want to change something about the article before it's approved for the aggregator they can contact the authors or ask someone to review it or whatever. Just make it accessible.

rhelz 3y ago
Whether or not your papers pass peer review—-and which journals it is published in—are important criteria for hiring, tenure, whether your grants are funded, etc. If you get rid of peer review, it’s not science. It’s just a vanity press.
- bumby 3y ago
  >If you get rid of peer review, it’s not science. It’s just a vanity press. If you believe replicability is central to science, the current paradigm doesn't necessarily converge on science either. And when people are graded on how many publications they garner, it borders on turning publication into a symbol of status rather than one of science.
  - rhelz 3y ago
    > If you believe replicability is central to science. I do believe that, but it doesn’t matter what anybody believes, replicatable experiments and results, which peers can review and agree on, are the soul of science. Without that it’s not science, it’s just creative writing.
    - 3y ago

PaulHoule 3y ago

And also has limited value as we practice it today. A useful review would involve: (a) "This paper won't be accepted by Cochrane for meta analysis", "N=20 get out of here", ... (b) Researchers provided their data files and Jupyter notebooks, the reviewer got them to run (c) Reviewers attempt their own analysis for at least some of the data (think of the model of accounting where auditors look at a sample of the books) (d) Reviewers come visit the lab and take a look at the apparatus (e) Something like a PhD defense (f) Summarize/formalize discussion of a paper that happens at a conference or online venue (g) 5 months, 5 years, or 40 years later (once in my case) somebody goes through the math line-by-line and finds a mistake on equation 178 that propagates and affects the rest of the calculation. This knowledge ought to be captured and "stapled" to the paper. I wouldn't say peer review is useless, I think it did improve papers I wrote a little but reviewers do not put enough effort into to reliably catch validity problems. If you believe in meta-analysis, which you should, read this book https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1114087/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1114087/ the first one (a) is really important because the overwhelming majority of papers are excluded in Cochrane reviews for most medical topics. If 80-90% of papers in some fields are not suitable for inference that raises tough questions such as "Why does work like this get published?" and "Why does this work get funded?" If we got half as many papers but 80-90% of them were usable that would be a revolution.

rhelz 3y ago
> If you believe in meta-analysis, which you should, I view meta-analysis as being like those mortgage-backed securities which crashed the world back in 2008. I mean, yeah, theoretically, a bond whose yield is a weighted average of mortgages should be less risky than any one of the mortgages in it. But....the devil is in the details. When I start seeing meta-analysis which claim that, e.g., masks don't protect you from a respiratory illness which spreads by coughing and sneezing, or that vaccines are at best worthless and at worse cause autism, wellll..... ....I have to conclude that garbage-in, garbage-out.
rhelz 3y ago
> I wouldn't say peer review is useless, I think it did improve papers I wrote a little Let's get back to basics here. What is the whole point of doing things scientifically in the first place? I.e. what is so special about the combination of 1) communicable descriptions of experiments, which can be re-run by anybody, and 2) the performing of said experiments and reporting the results, and 3) having others review and reproduce the results? Why are those 3 practices soooo special? Its because they are designed to either a) persuade everybody to agreement with you that your results are valid, or b) persuade you that you are wrong. I.e.. the scientific method is uniquely designed for the purpose of compelling agreement among peers. Its the 2nd best method we have of reaching consensus (the first best being the methods of proof used in mathematics. ) This is important because even if you make a new gee-whiz discovery, its useless unless other people agree that it is real. And unlike politics, or a beauty contest, or an aristocracy, it doesn't matter who you are, if your results are observable and reproducible, other scientists will agree with you, and your discovery has some chance of being generally useful. And that's why they are anonymous--its supposed to cut through the biases and prejudices which, alas, sometimes prevent us from coming to consensus. The point of peer review is to get a read on whether or not the author of the paper has justified their claims in enough detail that the results are likely to compile agreement. Yeah, we see if 3 or 4 people can agree on it, before asking everybody to agree on it. Whoda thunk? Point being, peer review isn't to help you improve your paper. > reviewers do not put enough effort into to reliably catch validity problems. Yes. This. Any why don't they? Because there's only so much work you can expect somebody to do for free. Lets pay them.