4 ms·

In practice, peer review isn't even that. Most referees are not double-checking your statistical analysis. What they focus on is whether the research is interes

by aggie 4y ago

In practice, peer review isn't even that. Most referees are not double-checking your statistical analysis. What they focus on is whether the research is interesting and has it appropriately considered relevant literature. Even that is not always done carefully. Here is an argument that peer review is basically a failed experiment: https://experimentalhistory.substack.com/p/the-rise-and-fall-of-peer-review https://experimentalhistory.substack.com/p/the-rise-and-fall...

dadrian 4y ago
I think we're saying the same thing. You're trusting the statistical analysis done by the author. When I peer review, I check to make sure that the conclusions the paper draws are in line with what it says statistical analysis was, and I check to make sure that the analysis used is the right one, but I don't check their math. And also, I'm not a statistician, so I'm not authoritative on the full space of statistical analysis, anyway.
- nieto915 4y ago
  who really has all of the hours in a day to do it too. I think it'll all come down to the journal's integrity. academia will have to learn to negotiate the premiums of their library subscriptions for better editing/curation for each journal they subscribe to. its already pay to play, might as well get your money's worth.
  - jhbadger 4y ago
    And often it is impossible to replicate the results as a reviewer even without fraud being involved because at least in studies involving DNA sequences, these sequences are generally only added to Genbank (or EMBL, or whatever repository depending on country) when a paper is accepted and so reviewers don't have access to it.
  - kuhewa 4y ago
    I'm burnt out on reviews, I did maybe ten last year but never spend less than a day on each and sometimes two. In my papers every reference fits to the best of my knowledge (knowledge informed by actually reading said cited material) and i follow best practices in statistical analyses, so I hold other papers to the same standard. Just an unsustainable standard to hold. I'd like to see journals pay statisticians to focus on methodology, I'm not sure if honourariums are a solution for peer review though. It would help me justify spending the time to do them but I reckon there would be a subset of hyperprolific reviewers half assing it more than ever to maximise the income per time spent.
wolverine876 4y ago
It's imperfect, but what better system do we replace it with? It's easy to critique; it's hard to solve problems. Just like it's easy to find problems in any research, but hard to research and publish.
- jostmey 4y ago
  The peer-review system is antiquated, developed before the internet and powerpoint presentations. Anything that facilitates interaction between authors and other researchers will be a vast improvement. Peer-review doesn't catch fraud and is sometimes a political process. I've found the best corrections I've gotten is after posting pre-prints to online forums. I suggest that commenting on pre-prints is better than peer-review pre-publication. I imagine a ranking system could highlight comments from trusted reviewers. Studies that no one wants to review were probably never going to be read anyway, and so there was never any reason to review these studies anyway.
  - a_bonobo 4y ago
    Yeah that's post-publication peer review, which I tend to gravitate towards myself; un-peer-reviewed papers in my field, posted on biorxiv, are generally of pretty high quality. (this might change as preprinting becomes a fully established route in biology) Any peer review added to that might improve some things, but is it worth it to add ~6 months to the publication timeline for a marginally improved manuscript? I'd say no.
- BurningFrog 4y ago
  Widespread peer review is only ~50 years old. Science did just fine before it, as it will after it's phased out.
  - jhbadger 4y ago
    Well, it was around but inconsistently applied. Einstein famously got angry when his work on gravity waves in 1935 was sent out to review ("I sent my paper to be published, not to be reviewed!"), and Watson & Crick's 1953 paper was going to be sent out to review but their boss, Nobel Laureate Laurence Bragg, phoned up the editor of Nature and said they needed to get it published ASAP.
ouid 4y ago
Like many human inventions, it started out good and got worse as people got better at exploiting the flaws in the institution. Those exploits eventually get bad enough that they can escalate to creating new flaws to be exploited until the whole thing is captured. Most institutions are somewhere along that path. My personal philosophy about such things is to think of Hanlon's razor as a boundary condition. The longer an institution has been around, the more likely the incompetence is actually just well disguised malice.
TheSpiceIsLife 4y ago
Science theatre
fsckboy 4y ago
In practice, peer review isn't even that. As fields of science have grown more specific, qualified reviewers are peers who are your competition, where you are already in their club or somebody who needs to be kept out. You are not going to be helped in any science that threatens their funding. There are a lot of interests vested in every instance of medical research fraud.

by aggie 4y ago

dadrian 4y ago

I think we're saying the same thing. You're trusting the statistical analysis done by the author. When I peer review, I check to make sure that the conclusions the paper draws are in line with what it says statistical analysis was, and I check to make sure that the analysis used is the right one, but I don't check their math. And also, I'm not a statistician, so I'm not authoritative on the full space of statistical analysis, anyway.

nieto915 4y ago
who really has all of the hours in a day to do it too. I think it'll all come down to the journal's integrity. academia will have to learn to negotiate the premiums of their library subscriptions for better editing/curation for each journal they subscribe to. its already pay to play, might as well get your money's worth.
- jhbadger 4y ago
  And often it is impossible to replicate the results as a reviewer even without fraud being involved because at least in studies involving DNA sequences, these sequences are generally only added to Genbank (or EMBL, or whatever repository depending on country) when a paper is accepted and so reviewers don't have access to it.
- kuhewa 4y ago
  I'm burnt out on reviews, I did maybe ten last year but never spend less than a day on each and sometimes two. In my papers every reference fits to the best of my knowledge (knowledge informed by actually reading said cited material) and i follow best practices in statistical analyses, so I hold other papers to the same standard. Just an unsustainable standard to hold. I'd like to see journals pay statisticians to focus on methodology, I'm not sure if honourariums are a solution for peer review though. It would help me justify spending the time to do them but I reckon there would be a subset of hyperprolific reviewers half assing it more than ever to maximise the income per time spent.

wolverine876 4y ago

It's imperfect, but what better system do we replace it with? It's easy to critique; it's hard to solve problems. Just like it's easy to find problems in any research, but hard to research and publish.

jostmey 4y ago
The peer-review system is antiquated, developed before the internet and powerpoint presentations. Anything that facilitates interaction between authors and other researchers will be a vast improvement. Peer-review doesn't catch fraud and is sometimes a political process. I've found the best corrections I've gotten is after posting pre-prints to online forums. I suggest that commenting on pre-prints is better than peer-review pre-publication. I imagine a ranking system could highlight comments from trusted reviewers. Studies that no one wants to review were probably never going to be read anyway, and so there was never any reason to review these studies anyway.
- a_bonobo 4y ago
  Yeah that's post-publication peer review, which I tend to gravitate towards myself; un-peer-reviewed papers in my field, posted on biorxiv, are generally of pretty high quality. (this might change as preprinting becomes a fully established route in biology) Any peer review added to that might improve some things, but is it worth it to add ~6 months to the publication timeline for a marginally improved manuscript? I'd say no.
BurningFrog 4y ago
Widespread peer review is only ~50 years old. Science did just fine before it, as it will after it's phased out.
- jhbadger 4y ago
  Well, it was around but inconsistently applied. Einstein famously got angry when his work on gravity waves in 1935 was sent out to review ("I sent my paper to be published, not to be reviewed!"), and Watson & Crick's 1953 paper was going to be sent out to review but their boss, Nobel Laureate Laurence Bragg, phoned up the editor of Nature and said they needed to get it published ASAP.

ouid 4y ago

Like many human inventions, it started out good and got worse as people got better at exploiting the flaws in the institution. Those exploits eventually get bad enough that they can escalate to creating new flaws to be exploited until the whole thing is captured. Most institutions are somewhere along that path. My personal philosophy about such things is to think of Hanlon's razor as a boundary condition. The longer an institution has been around, the more likely the incompetence is actually just well disguised malice.

TheSpiceIsLife 4y ago

Science theatre

fsckboy 4y ago

In practice, peer review isn't even that. As fields of science have grown more specific, qualified reviewers are peers who are your competition, where you are already in their club or somebody who needs to be kept out. You are not going to be helped in any science that threatens their funding. There are a lot of interests vested in every instance of medical research fraud.