5 ms·

Alternative take: A system designed to detect bad research failed to detect blatant falsehoods, algorithmically generated noise, rabid propaganda and other seem

Alternative take: A system designed to detect bad research failed to detect blatant falsehoods, algorithmically generated noise, rabid propaganda and other seemingly easy things to detect. That PhDs worked on this doesn’t mean the system didn’t fail spectacularly.

ModernMech 8y ago
> doesn’t mean the system didn’t fail spectacularly. You're mischaracterizing the system; it doesn't end with publication. After publication, the science goes out to a wider audience of scientists who can build on the paper or refute it if they find it's wrong. Sometimes this process can take a long time and many cycles of publicans, but that's just the nature of science. We have to stop pretending like the peer review process is some kind of magical filter of true science, or that it was ever intended to be as such. That notion leads to the faulty idea that if it's published, it's true. Junk science gets published, even if it's not disingenuous, and that's probably true no matter how great you make the peer review system.
- malvosenior 8y ago
  I'm not saying that peer review is the end all be all of truth, but it does serve some purpose. And in this case it didn't provide any value at all.
- pochamago 8y ago
  What is the point of journals if quite literally anything can be published in them? Isn't the point that they serve as a basic filter for obvious noise? Shouldn't we be critical of a peer review process that doesn't function at even the most basic level?
  - PeterisP 8y ago
    A filter that does filter out the vast majority of obvious noise is very useful even if it lets something through. And the current peer-review does that, it throws out almost all of the spam; it pushes useful work of lower quality/impact down to less selective venues (generally with more specific subfields, so that I can read impactful work of domain X and less impactful work of subdomain X.2 but not X.1 and X.3); and even for accepted papers the process often results in significant improvements to them as a result of the reviews. Explicit fraud isn't that common, since it does have risks to the submitter that far outweigh the (limited) benefits of getting an extra publication on your CV.
  - ModernMech 8y ago
    > quite literally anything can be published I've got a stack of rejected papers that beg to differ. > Isn't the point that they serve as a basic filter for obvious noise? Yeah, and obvious noise is usually rejected. But sometimes it is not so obvious what the noise is, and that's why the scientific process is bigger than the peer review system. > Isn't the point that they serve as a basic filter for obvious noise? Totally, there is a lot to fix about it. But if you really want to put your effort into changing the system, I don't think you should focus on peer review. Destroying the business model of journals is where I would start.
- HarryHirsch 8y ago
  Why is this being downvoted? This is a description of the "invisible boot" as Broad & Wade call it.
UncleMeat 8y ago
Peer Review is not set up to detect fraudulent research and attempts to do so are fraught with risk, since established researchers often have a hard time believing in results from new paradigms. What the heck is an author supposed to do if a reviewer says "I think these numbers are made up"? The purpose of peer review is to decide whether a paper is useful to the community, whether the presentation is good enough that the community can precisely understand what happened, and whether it has methodological issues that the authors may have missed. But we always approach reviews with a charitable lens, even when reviewing for extremely low-acceptance outlets. The process that detects the bad research comes afterwards, as it should.
- malvosenior 8y ago
  > The purpose of peer review is to decide whether a paper is useful to the community, whether the presentation is good enough that the community can precisely understand what happened, and whether it has methodological issues that the authors may have missed. It clearly failed to do that in this case.
  - UncleMeat 8y ago
    Ok? That somebody with a year of free time and a pile of fraudulent attempts can get through sometimes is not exactly a surprise. A system that prevents what the authors did here would necessarily reject a huge amount of useful research.

ModernMech 8y ago

> doesn’t mean the system didn’t fail spectacularly. You're mischaracterizing the system; it doesn't end with publication. After publication, the science goes out to a wider audience of scientists who can build on the paper or refute it if they find it's wrong. Sometimes this process can take a long time and many cycles of publicans, but that's just the nature of science. We have to stop pretending like the peer review process is some kind of magical filter of true science, or that it was ever intended to be as such. That notion leads to the faulty idea that if it's published, it's true. Junk science gets published, even if it's not disingenuous, and that's probably true no matter how great you make the peer review system.

malvosenior 8y ago
I'm not saying that peer review is the end all be all of truth, but it does serve some purpose. And in this case it didn't provide any value at all.
pochamago 8y ago
What is the point of journals if quite literally anything can be published in them? Isn't the point that they serve as a basic filter for obvious noise? Shouldn't we be critical of a peer review process that doesn't function at even the most basic level?
- PeterisP 8y ago
  A filter that does filter out the vast majority of obvious noise is very useful even if it lets something through. And the current peer-review does that, it throws out almost all of the spam; it pushes useful work of lower quality/impact down to less selective venues (generally with more specific subfields, so that I can read impactful work of domain X and less impactful work of subdomain X.2 but not X.1 and X.3); and even for accepted papers the process often results in significant improvements to them as a result of the reviews. Explicit fraud isn't that common, since it does have risks to the submitter that far outweigh the (limited) benefits of getting an extra publication on your CV.
- ModernMech 8y ago
  > quite literally anything can be published I've got a stack of rejected papers that beg to differ. > Isn't the point that they serve as a basic filter for obvious noise? Yeah, and obvious noise is usually rejected. But sometimes it is not so obvious what the noise is, and that's why the scientific process is bigger than the peer review system. > Isn't the point that they serve as a basic filter for obvious noise? Totally, there is a lot to fix about it. But if you really want to put your effort into changing the system, I don't think you should focus on peer review. Destroying the business model of journals is where I would start.
HarryHirsch 8y ago
Why is this being downvoted? This is a description of the "invisible boot" as Broad & Wade call it.

UncleMeat 8y ago

Peer Review is not set up to detect fraudulent research and attempts to do so are fraught with risk, since established researchers often have a hard time believing in results from new paradigms. What the heck is an author supposed to do if a reviewer says "I think these numbers are made up"? The purpose of peer review is to decide whether a paper is useful to the community, whether the presentation is good enough that the community can precisely understand what happened, and whether it has methodological issues that the authors may have missed. But we always approach reviews with a charitable lens, even when reviewing for extremely low-acceptance outlets. The process that detects the bad research comes afterwards, as it should.

malvosenior 8y ago
> The purpose of peer review is to decide whether a paper is useful to the community, whether the presentation is good enough that the community can precisely understand what happened, and whether it has methodological issues that the authors may have missed. It clearly failed to do that in this case.
- UncleMeat 8y ago
  Ok? That somebody with a year of free time and a pile of fraudulent attempts can get through sometimes is not exactly a surprise. A system that prevents what the authors did here would necessarily reject a huge amount of useful research.