3 ms·

Good. Not catching a simple multiplication error is a terrible indictment of their peer review process. On the other hand, peer review is doomed and getting mor

by lores 2y ago

Good. Not catching a simple multiplication error is a terrible indictment of their peer review process. On the other hand, peer review is doomed and getting more doomed by the day, and I expect more and more journals will fail if the quality criteria get fairly applied.

crazygringo 2y ago
I never thought of peer review as being meant to verify calculations. But rather to ensure the experimental methodology was sound, it cites the appropriate previous work, that it is significant enough, etc. I actually think that calculation verification and statistical significance verification is a great idea, but shouldn't be done by "peers", but rather someone who specializes only in that. More of a type of auditor.
- burkaman 2y ago
  I don't think peer reviewers need to be redoing every calculation, but you might expect them to be familiar enough with the field that "toxins in kitchen utensils are close to the EPA limit" would be surprising enough that they'd question it and double check it.
  - bpodgursky 2y ago
    How many researchers do you think are working on cookware toxins? "Peer" is used broadly here. The article made waves because it didn't surprise anyone, because in fact a ton of household staples are full of toxins (spices are full of lead, etc).
    - pessimizer 2y ago
      > The article made waves because it didn't surprise anyone So the reason it made waves is because it confirmed everyone's previous biases. Which is why people defend it even when that "confirmation" was based on an error of simple arithmetic. Reminds me of a bunch of studies that were taken out by Excel errors, like that economics failure about national debt going higher than GDP being some sort of trigger for societal failure. The people who boosted that study were the people who were saying the same thing, with no study, for no other reason than that the numbers are of a similar magnitude. After it was shown that the study showed no such thing, its supporters insisted that it was still true because of course it was true. https://stanfordreview.org/clarifying-the-implications-of-the-reinhart-rogoff-excel-error/ https://stanfordreview.org/clarifying-the-implications-of-th...
    - burkaman 2y ago
      I don't agree that it didn't surprise anyone. Flame retardants are surprising, black plastic being worse than others is surprising, common household tools nearly at the legal toxin limit is surprising. Even the general idea of plastic being potentially bad for you is still new information for a lot of people. Your comment is the first time I've ever heard anything about lead in spices, and I read a lot of news. In my experience, very few things are actually "common knowledge", it just seems that way when everyone in our particular social circles knows something.
      - Dylan16807 2y ago
        > common household tools nearly at the legal toxin limit is surprising Personally, I would expect a lot of products to be near a lot of limits for various toxins. Especially around food, my expectation is for limits to be safe with a nice-sized buffer, sometimes too big of a buffer. The point of having limits is that products inside them are fine.
coldtea 2y ago
>Not catching a simple multiplication error is a terrible indictment of their peer review process Most papers are full of errors nobody ever catches (or cares to catch).
- _yb2s 2y ago
  Yes, as an academic scientist when I occasionally need to actually reproduce something from published literature to use in my own research- I almost always find at least half a dozen errors, even in a well respected journal. The only way to really find such errors is to painstakingly reproduce the work, and that is not part of the publication process- it would double the time and cost.
wyager 2y ago
"Peer review" has been next to useless in most fields for the last 50-100 years at least. We need a new epistemic benchmark.
PunchTornado 2y ago
I reviewed articles and never verified their calculations because it is not part of what a reviewer should do. A reviewer should never try and replicate the experiment or crunch numbers. It should look at the methodology, background, significance, possible enhancements etc.
mrguyorama 2y ago
Peer review is not an important part of the scientific method actually. It's not even necessary. There was no peer review for most of what we now take for granted in science. The Journal Nature did not make peer review standard until 1973. Science is about making reality the arbiter of truth. What is true is what you can demonstrate in an experiment pretty much anywhere. Science is self-correcting not because other scientists take a look at your work before it gets published, but because a theory that is not backed up by reality will be harder to build on, make worse predictions, and generally not be useful for actual work. It doesn't matter how much "hype" built up around LK-99 for example, because you could not build anything out of it, because it didn't work as claimed, and no amount of belief can overcome that. Science is self correcting because the universe is not convinced by rhetoric, and will only ever follow it's own rules. Tellingly, scientists who have been around for a while do not consider a published paper as especially meaningful. They do not consider peer review as vouching for a paper's correctness. They have seen hundreds of papers pass peer review and make it into Science and Nature and turn out to be bunk. No scientific process will be perfect, so some amount of papers should be expected to be wrong, no matter how genuinely careful and perfect the scientists who wrote it were. It's really sad how over-indexed the lay-public has gotten on peer review. It's just not how science works. It can still be somewhat useful for more popular journals who have to pick and choose what to publish anyway, but it's only useful if the peer reviewers are themselves domain experts in a rigorous and formalized scientific discipline.
_yb2s 2y ago
As an academic scientist that does peer review- I would never consider redoing math unless the conclusion seemed suspicious. Peer reviewers don’t have a lot of time and need to assume good faith and careful basic competency from the authors. We spend realistically about an hour to review a study that might have taken a whole team of authors half a decade. Mostly we are looking to confirm that the work is presented clearly, and that the evidence is sufficient to support the conclusions. Peer review is little more than proofreading from an unbiased 3rd party.
userbinator 2y ago
In the case of these journals, "peer" simply means "someone who shares the same agenda", and they won't care about the science or even the numerical results as long as the conclusion agrees with their ideology. The extremist envrionmentalists have been pushing their idiotic agenda for far too long, and it's good to see the sane people are now pushing back.

by lores 2y ago

crazygringo 2y ago

I never thought of peer review as being meant to verify calculations. But rather to ensure the experimental methodology was sound, it cites the appropriate previous work, that it is significant enough, etc. I actually think that calculation verification and statistical significance verification is a great idea, but shouldn't be done by "peers", but rather someone who specializes only in that. More of a type of auditor.

burkaman 2y ago
I don't think peer reviewers need to be redoing every calculation, but you might expect them to be familiar enough with the field that "toxins in kitchen utensils are close to the EPA limit" would be surprising enough that they'd question it and double check it.
- bpodgursky 2y ago
  How many researchers do you think are working on cookware toxins? "Peer" is used broadly here. The article made waves because it didn't surprise anyone, because in fact a ton of household staples are full of toxins (spices are full of lead, etc).
  - pessimizer 2y ago
    > The article made waves because it didn't surprise anyone So the reason it made waves is because it confirmed everyone's previous biases. Which is why people defend it even when that "confirmation" was based on an error of simple arithmetic. Reminds me of a bunch of studies that were taken out by Excel errors, like that economics failure about national debt going higher than GDP being some sort of trigger for societal failure. The people who boosted that study were the people who were saying the same thing, with no study, for no other reason than that the numbers are of a similar magnitude. After it was shown that the study showed no such thing, its supporters insisted that it was still true because of course it was true. https://stanfordreview.org/clarifying-the-implications-of-the-reinhart-rogoff-excel-error/ https://stanfordreview.org/clarifying-the-implications-of-th...
  - burkaman 2y ago
    I don't agree that it didn't surprise anyone. Flame retardants are surprising, black plastic being worse than others is surprising, common household tools nearly at the legal toxin limit is surprising. Even the general idea of plastic being potentially bad for you is still new information for a lot of people. Your comment is the first time I've ever heard anything about lead in spices, and I read a lot of news. In my experience, very few things are actually "common knowledge", it just seems that way when everyone in our particular social circles knows something.
    - Dylan16807 2y ago
      > common household tools nearly at the legal toxin limit is surprising Personally, I would expect a lot of products to be near a lot of limits for various toxins. Especially around food, my expectation is for limits to be safe with a nice-sized buffer, sometimes too big of a buffer. The point of having limits is that products inside them are fine.

coldtea 2y ago

>Not catching a simple multiplication error is a terrible indictment of their peer review process Most papers are full of errors nobody ever catches (or cares to catch).

_yb2s 2y ago
Yes, as an academic scientist when I occasionally need to actually reproduce something from published literature to use in my own research- I almost always find at least half a dozen errors, even in a well respected journal. The only way to really find such errors is to painstakingly reproduce the work, and that is not part of the publication process- it would double the time and cost.

wyager 2y ago

"Peer review" has been next to useless in most fields for the last 50-100 years at least. We need a new epistemic benchmark.

PunchTornado 2y ago

I reviewed articles and never verified their calculations because it is not part of what a reviewer should do. A reviewer should never try and replicate the experiment or crunch numbers. It should look at the methodology, background, significance, possible enhancements etc.

mrguyorama 2y ago

Peer review is not an important part of the scientific method actually. It's not even necessary. There was no peer review for most of what we now take for granted in science. The Journal Nature did not make peer review standard until 1973. Science is about making reality the arbiter of truth. What is true is what you can demonstrate in an experiment pretty much anywhere. Science is self-correcting not because other scientists take a look at your work before it gets published, but because a theory that is not backed up by reality will be harder to build on, make worse predictions, and generally not be useful for actual work. It doesn't matter how much "hype" built up around LK-99 for example, because you could not build anything out of it, because it didn't work as claimed, and no amount of belief can overcome that. Science is self correcting because the universe is not convinced by rhetoric, and will only ever follow it's own rules. Tellingly, scientists who have been around for a while do not consider a published paper as especially meaningful. They do not consider peer review as vouching for a paper's correctness. They have seen hundreds of papers pass peer review and make it into Science and Nature and turn out to be bunk. No scientific process will be perfect, so some amount of papers should be expected to be wrong, no matter how genuinely careful and perfect the scientists who wrote it were. It's really sad how over-indexed the lay-public has gotten on peer review. It's just not how science works. It can still be somewhat useful for more popular journals who have to pick and choose what to publish anyway, but it's only useful if the peer reviewers are themselves domain experts in a rigorous and formalized scientific discipline.

_yb2s 2y ago

As an academic scientist that does peer review- I would never consider redoing math unless the conclusion seemed suspicious. Peer reviewers don’t have a lot of time and need to assume good faith and careful basic competency from the authors. We spend realistically about an hour to review a study that might have taken a whole team of authors half a decade. Mostly we are looking to confirm that the work is presented clearly, and that the evidence is sufficient to support the conclusions. Peer review is little more than proofreading from an unbiased 3rd party.

userbinator 2y ago

In the case of these journals, "peer" simply means "someone who shares the same agenda", and they won't care about the science or even the numerical results as long as the conclusion agrees with their ideology. The extremist envrionmentalists have been pushing their idiotic agenda for far too long, and it's good to see the sane people are now pushing back.