7 ms·

As a reminder, peer review is not designed adversarially. It is not supposed to catch people who are fraudulent before they publish---that's the point of replic

As a reminder, peer review is not designed adversarially. It is not supposed to catch people who are fraudulent before they publish---that's the point of replication (which happens rarily). Peer review is designed to ensure a the publication as a packaged work of science describes a valid experiment. But you're necessarily assuming that the authors did what they said they did. As a peer reviewer, you're ensuring that what the authors said they did constitutes valid science.

aggie 4y ago
In practice, peer review isn't even that. Most referees are not double-checking your statistical analysis. What they focus on is whether the research is interesting and has it appropriately considered relevant literature. Even that is not always done carefully. Here is an argument that peer review is basically a failed experiment: https://experimentalhistory.substack.com/p/the-rise-and-fall-of-peer-review https://experimentalhistory.substack.com/p/the-rise-and-fall...
- dadrian 4y ago
  I think we're saying the same thing. You're trusting the statistical analysis done by the author. When I peer review, I check to make sure that the conclusions the paper draws are in line with what it says statistical analysis was, and I check to make sure that the analysis used is the right one, but I don't check their math. And also, I'm not a statistician, so I'm not authoritative on the full space of statistical analysis, anyway.
  - nieto915 4y ago
    who really has all of the hours in a day to do it too. I think it'll all come down to the journal's integrity. academia will have to learn to negotiate the premiums of their library subscriptions for better editing/curation for each journal they subscribe to. its already pay to play, might as well get your money's worth.
    - jhbadger 4y ago
      And often it is impossible to replicate the results as a reviewer even without fraud being involved because at least in studies involving DNA sequences, these sequences are generally only added to Genbank (or EMBL, or whatever repository depending on country) when a paper is accepted and so reviewers don't have access to it.
    - kuhewa 4y ago
      I'm burnt out on reviews, I did maybe ten last year but never spend less than a day on each and sometimes two. In my papers every reference fits to the best of my knowledge (knowledge informed by actually reading said cited material) and i follow best practices in statistical analyses, so I hold other papers to the same standard. Just an unsustainable standard to hold. I'd like to see journals pay statisticians to focus on methodology, I'm not sure if honourariums are a solution for peer review though. It would help me justify spending the time to do them but I reckon there would be a subset of hyperprolific reviewers half assing it more than ever to maximise the income per time spent.
- wolverine876 4y ago
  It's imperfect, but what better system do we replace it with? It's easy to critique; it's hard to solve problems. Just like it's easy to find problems in any research, but hard to research and publish.
  - jostmey 4y ago
    The peer-review system is antiquated, developed before the internet and powerpoint presentations. Anything that facilitates interaction between authors and other researchers will be a vast improvement. Peer-review doesn't catch fraud and is sometimes a political process. I've found the best corrections I've gotten is after posting pre-prints to online forums. I suggest that commenting on pre-prints is better than peer-review pre-publication. I imagine a ranking system could highlight comments from trusted reviewers. Studies that no one wants to review were probably never going to be read anyway, and so there was never any reason to review these studies anyway.
    - a_bonobo 4y ago
      Yeah that's post-publication peer review, which I tend to gravitate towards myself; un-peer-reviewed papers in my field, posted on biorxiv, are generally of pretty high quality. (this might change as preprinting becomes a fully established route in biology) Any peer review added to that might improve some things, but is it worth it to add ~6 months to the publication timeline for a marginally improved manuscript? I'd say no.
  - BurningFrog 4y ago
    Widespread peer review is only ~50 years old. Science did just fine before it, as it will after it's phased out.
    - jhbadger 4y ago
      Well, it was around but inconsistently applied. Einstein famously got angry when his work on gravity waves in 1935 was sent out to review ("I sent my paper to be published, not to be reviewed!"), and Watson & Crick's 1953 paper was going to be sent out to review but their boss, Nobel Laureate Laurence Bragg, phoned up the editor of Nature and said they needed to get it published ASAP.
- ouid 4y ago
  Like many human inventions, it started out good and got worse as people got better at exploiting the flaws in the institution. Those exploits eventually get bad enough that they can escalate to creating new flaws to be exploited until the whole thing is captured. Most institutions are somewhere along that path. My personal philosophy about such things is to think of Hanlon's razor as a boundary condition. The longer an institution has been around, the more likely the incompetence is actually just well disguised malice.
- TheSpiceIsLife 4y ago
  Science theatre
- fsckboy 4y ago
  In practice, peer review isn't even that. As fields of science have grown more specific, qualified reviewers are peers who are your competition, where you are already in their club or somebody who needs to be kept out. You are not going to be helped in any science that threatens their funding. There are a lot of interests vested in every instance of medical research fraud.
jltsiren 4y ago
I don't think peer review is about ensuring anything. It's about spending some time to improve the quality of the paper to save time for everyone else. It's more about providing editorial services than about the reported work itself. A research paper is supposed to be a honest report of best efforts to study a topic. If it's not, that's a problem that can't be solved by having a few people spending a few hours with the report. The paper is not the final word on anything anyway. If you read a paper expecting to learn something about the world, you are doing something wrong. As a reviewer, you determine whether the paper is interesting and relevant to the venue. You report any issues you spot that should be corrected and any things you believe that could be improved. And if you get any ideas you feel like sharing, you may share them as well. And that's it.
- ifyoubuildit 4y ago
  > The paper is not the final word on anything anyway. If you read a paper expecting to learn something about the world, you are doing something wrong. Can you expand on this? As a layperson, I'm wondering what would you read a paper for at all if not to learn something about the world? What's the point otherwise?
  - jltsiren 4y ago
    Papers report very early stages of knowledge formation. They typically contain an argument: what was done, why, what was expected to happen, what actually happened, how did the authors interpret that, and why. Rather than describing the world directly, a paper describes a belief about the world, and justifications for the belief. Published results are often contradictory, because individual papers are unreliable. Something may have gone subtly wrong, the interpretation could lack nuances, some key understanding may still be missing, or the authors may have just been unlucky. When an expert reads many papers on related topics, the arguments shape their beliefs. Eventually a scientific consensus may emerge, which is the next (but still an insufficient) step towards reliable knowledge.
    - tptacek 4y ago
      This is something I bookmarked (in part because I'm an admirer, but also because it made sense and squared with my limited experience of academic peer review): https://twitter.com/TheSavageInMan/status/1083503677157961728 https://twitter.com/TheSavageInMan/status/108350367715796172... I think people get in trouble when they think academic publication has value in itself. Ideas become valuable when people care about them. Publication is one particular path to reach a community of readers but it doesn’t make your ideas matter.
NtochkaNzvanova 4y ago
Disagree. From my experience as a scientist there isn't a manual that says what peer review is and isn't supposed to be. Of course, some journals have such guidelines (not that all reviewers read them), and the editors who assign the reviews are free to ignore a rogue reviewer. But there is a set of explicit or tacit standards of good science, and any of it is fair game in peer review: 1. Validity of the experiment 2. Interestingness / novelty of the work 3. Appropriate choice of methods and correctness / believability of the results 4. Signs of outright fraud (suspicious figures, etc.) Absolutely if I notice some weird Photoshop artifacts in a figure, or some other obvious sign of fake data, I'm going to call that out. I'm probably an outlier on this next one, but I've even been tempted to reject articles just for having egregiously, unreadably bad writing. I know this will be regarded as bias against people whose first language is not English, but if the writing is so bad that it stands in the way of making sense of the article, and the authors can't be bothered to get a decent editor, it's not a worthy contribution to the (English language) scientific literature.
- tptacek 4y ago
  I don't think anybody is denying that if something jumps out at you as hinky in a paper, you're going to note it --- in fact, if you believe the paper is fabricated, you'll probably do more than that, and raise a stink with the chairs of the PC as well. The problem people have conceptualizing peer review is that reviewers can't reliably spot stuff; they simply don't have time to do it, and it isn't the premise of the exercise.
lr1970 4y ago
> As a reminder, peer review is not designed adversarially. I beg to differ. Peer review is an adversarial process. The author(s) of the paper are making statements and report findings proving them by logic or/and data and analysis. The priors while reviewing a science paper is "wrong" until proven "correct". This also covers accuracy and veracity of the data and analysis. The only thing a peer review is not is assigning intent or blame. It is not the job of a reviewer to look for fraud when simple incompetence could explain it. But after reading this article I will add fraud dimension to a list of fallacies I am looking for while peer-reviewing a manuscript.

aggie 4y ago

In practice, peer review isn't even that. Most referees are not double-checking your statistical analysis. What they focus on is whether the research is interesting and has it appropriately considered relevant literature. Even that is not always done carefully. Here is an argument that peer review is basically a failed experiment: https://experimentalhistory.substack.com/p/the-rise-and-fall-of-peer-review https://experimentalhistory.substack.com/p/the-rise-and-fall...

dadrian 4y ago
I think we're saying the same thing. You're trusting the statistical analysis done by the author. When I peer review, I check to make sure that the conclusions the paper draws are in line with what it says statistical analysis was, and I check to make sure that the analysis used is the right one, but I don't check their math. And also, I'm not a statistician, so I'm not authoritative on the full space of statistical analysis, anyway.
- nieto915 4y ago
  who really has all of the hours in a day to do it too. I think it'll all come down to the journal's integrity. academia will have to learn to negotiate the premiums of their library subscriptions for better editing/curation for each journal they subscribe to. its already pay to play, might as well get your money's worth.
  - jhbadger 4y ago
    And often it is impossible to replicate the results as a reviewer even without fraud being involved because at least in studies involving DNA sequences, these sequences are generally only added to Genbank (or EMBL, or whatever repository depending on country) when a paper is accepted and so reviewers don't have access to it.
  - kuhewa 4y ago
    I'm burnt out on reviews, I did maybe ten last year but never spend less than a day on each and sometimes two. In my papers every reference fits to the best of my knowledge (knowledge informed by actually reading said cited material) and i follow best practices in statistical analyses, so I hold other papers to the same standard. Just an unsustainable standard to hold. I'd like to see journals pay statisticians to focus on methodology, I'm not sure if honourariums are a solution for peer review though. It would help me justify spending the time to do them but I reckon there would be a subset of hyperprolific reviewers half assing it more than ever to maximise the income per time spent.
wolverine876 4y ago
It's imperfect, but what better system do we replace it with? It's easy to critique; it's hard to solve problems. Just like it's easy to find problems in any research, but hard to research and publish.
- jostmey 4y ago
  The peer-review system is antiquated, developed before the internet and powerpoint presentations. Anything that facilitates interaction between authors and other researchers will be a vast improvement. Peer-review doesn't catch fraud and is sometimes a political process. I've found the best corrections I've gotten is after posting pre-prints to online forums. I suggest that commenting on pre-prints is better than peer-review pre-publication. I imagine a ranking system could highlight comments from trusted reviewers. Studies that no one wants to review were probably never going to be read anyway, and so there was never any reason to review these studies anyway.
  - a_bonobo 4y ago
    Yeah that's post-publication peer review, which I tend to gravitate towards myself; un-peer-reviewed papers in my field, posted on biorxiv, are generally of pretty high quality. (this might change as preprinting becomes a fully established route in biology) Any peer review added to that might improve some things, but is it worth it to add ~6 months to the publication timeline for a marginally improved manuscript? I'd say no.
- BurningFrog 4y ago
  Widespread peer review is only ~50 years old. Science did just fine before it, as it will after it's phased out.
  - jhbadger 4y ago
    Well, it was around but inconsistently applied. Einstein famously got angry when his work on gravity waves in 1935 was sent out to review ("I sent my paper to be published, not to be reviewed!"), and Watson & Crick's 1953 paper was going to be sent out to review but their boss, Nobel Laureate Laurence Bragg, phoned up the editor of Nature and said they needed to get it published ASAP.
ouid 4y ago
Like many human inventions, it started out good and got worse as people got better at exploiting the flaws in the institution. Those exploits eventually get bad enough that they can escalate to creating new flaws to be exploited until the whole thing is captured. Most institutions are somewhere along that path. My personal philosophy about such things is to think of Hanlon's razor as a boundary condition. The longer an institution has been around, the more likely the incompetence is actually just well disguised malice.
TheSpiceIsLife 4y ago
Science theatre
fsckboy 4y ago
In practice, peer review isn't even that. As fields of science have grown more specific, qualified reviewers are peers who are your competition, where you are already in their club or somebody who needs to be kept out. You are not going to be helped in any science that threatens their funding. There are a lot of interests vested in every instance of medical research fraud.

jltsiren 4y ago

I don't think peer review is about ensuring anything. It's about spending some time to improve the quality of the paper to save time for everyone else. It's more about providing editorial services than about the reported work itself. A research paper is supposed to be a honest report of best efforts to study a topic. If it's not, that's a problem that can't be solved by having a few people spending a few hours with the report. The paper is not the final word on anything anyway. If you read a paper expecting to learn something about the world, you are doing something wrong. As a reviewer, you determine whether the paper is interesting and relevant to the venue. You report any issues you spot that should be corrected and any things you believe that could be improved. And if you get any ideas you feel like sharing, you may share them as well. And that's it.

ifyoubuildit 4y ago
> The paper is not the final word on anything anyway. If you read a paper expecting to learn something about the world, you are doing something wrong. Can you expand on this? As a layperson, I'm wondering what would you read a paper for at all if not to learn something about the world? What's the point otherwise?
- jltsiren 4y ago
  Papers report very early stages of knowledge formation. They typically contain an argument: what was done, why, what was expected to happen, what actually happened, how did the authors interpret that, and why. Rather than describing the world directly, a paper describes a belief about the world, and justifications for the belief. Published results are often contradictory, because individual papers are unreliable. Something may have gone subtly wrong, the interpretation could lack nuances, some key understanding may still be missing, or the authors may have just been unlucky. When an expert reads many papers on related topics, the arguments shape their beliefs. Eventually a scientific consensus may emerge, which is the next (but still an insufficient) step towards reliable knowledge.
  - tptacek 4y ago
    This is something I bookmarked (in part because I'm an admirer, but also because it made sense and squared with my limited experience of academic peer review): https://twitter.com/TheSavageInMan/status/1083503677157961728 https://twitter.com/TheSavageInMan/status/108350367715796172... I think people get in trouble when they think academic publication has value in itself. Ideas become valuable when people care about them. Publication is one particular path to reach a community of readers but it doesn’t make your ideas matter.

NtochkaNzvanova 4y ago

Disagree. From my experience as a scientist there isn't a manual that says what peer review is and isn't supposed to be. Of course, some journals have such guidelines (not that all reviewers read them), and the editors who assign the reviews are free to ignore a rogue reviewer. But there is a set of explicit or tacit standards of good science, and any of it is fair game in peer review: 1. Validity of the experiment 2. Interestingness / novelty of the work 3. Appropriate choice of methods and correctness / believability of the results 4. Signs of outright fraud (suspicious figures, etc.) Absolutely if I notice some weird Photoshop artifacts in a figure, or some other obvious sign of fake data, I'm going to call that out. I'm probably an outlier on this next one, but I've even been tempted to reject articles just for having egregiously, unreadably bad writing. I know this will be regarded as bias against people whose first language is not English, but if the writing is so bad that it stands in the way of making sense of the article, and the authors can't be bothered to get a decent editor, it's not a worthy contribution to the (English language) scientific literature.

tptacek 4y ago
I don't think anybody is denying that if something jumps out at you as hinky in a paper, you're going to note it --- in fact, if you believe the paper is fabricated, you'll probably do more than that, and raise a stink with the chairs of the PC as well. The problem people have conceptualizing peer review is that reviewers can't reliably spot stuff; they simply don't have time to do it, and it isn't the premise of the exercise.

lr1970 4y ago

> As a reminder, peer review is not designed adversarially. I beg to differ. Peer review is an adversarial process. The author(s) of the paper are making statements and report findings proving them by logic or/and data and analysis. The priors while reviewing a science paper is "wrong" until proven "correct". This also covers accuracy and veracity of the data and analysis. The only thing a peer review is not is assigning intent or blame. It is not the job of a reviewer to look for fraud when simple incompetence could explain it. But after reading this article I will add fraud dimension to a list of fallacies I am looking for while peer-reviewing a manuscript.