5 ms·

Representative quote: > "I am so utterly unimpressed," wrote Jacob T. Levy, a political theorist at McGill University, "by the fact that an enterprise that rel

• throwaway5250 8y ago
  Representative quotes: "Is there any idea so outlandish that it won’t be published in a Critical/PoMo/Identity/‘Theory’ journal?" tweeted the Harvard psychologist Steven Pinker. "Three intrepid academics," wrote Yascha Mounk, an author and lecturer on government at Harvard, "just perpetrated a giant version of the Sokal Hoax, placing … fake papers in major academic journals. Call it Sokal Squared. The result is hilarious and delightful. It also showcases a serious problem with big parts of academia."
  • tptacek 8y ago
    Mounk didn't read the hoaxers writeup carefully enough to be a reliable judge; he gleefully tweeted two papers from the hoax that were rejected everywhere they were submitted, as evidence of the important problem the hoax uncovered --- this despite the fact that the hoaxers did in fact provide an accounting of what got accepted where (even though their writeup at the beginning of the piece misrepresented that accounting). From what Pinker wrote, who knows if he even read it?
• debacle 8y ago
  Shouldn't the default presupposition of any research be a lack of novelty?
  • Trombone12 8y ago
    I think generally people tend to assume that colleagues aren't dishonest, and mostly peer-review is supposed to catch honest mistakes. You could try working without any faith in your colleagues of course, but you are probably better of switching workplace if that sounds like reasonable behaviour to you.
    • debacle 8y ago
      What your describing sounds nothing like rigor.
      • UncleMeat 8y ago
        The rigor comes from the experiment and the future analysis by other researchers, not from the review. You are expecting something from peer review that never exists.
• the8472 8y ago
  > can be fooled some of the time 7⁄20 or 7⁄17, depending on how you want to count is concerning.
• malvosenior 8y ago
  Alternative take: A system designed to detect bad research failed to detect blatant falsehoods, algorithmically generated noise, rabid propaganda and other seemingly easy things to detect. That PhDs worked on this doesn’t mean the system didn’t fail spectacularly.
  • ModernMech 8y ago
    > doesn’t mean the system didn’t fail spectacularly. You're mischaracterizing the system; it doesn't end with publication. After publication, the science goes out to a wider audience of scientists who can build on the paper or refute it if they find it's wrong. Sometimes this process can take a long time and many cycles of publicans, but that's just the nature of science. We have to stop pretending like the peer review process is some kind of magical filter of true science, or that it was ever intended to be as such. That notion leads to the faulty idea that if it's published, it's true. Junk science gets published, even if it's not disingenuous, and that's probably true no matter how great you make the peer review system.
    • malvosenior 8y ago
      I'm not saying that peer review is the end all be all of truth, but it does serve some purpose. And in this case it didn't provide any value at all.
    • pochamago 8y ago
      What is the point of journals if quite literally anything can be published in them? Isn't the point that they serve as a basic filter for obvious noise? Shouldn't we be critical of a peer review process that doesn't function at even the most basic level?
      • PeterisP 8y ago
        A filter that does filter out the vast majority of obvious noise is very useful even if it lets something through. And the current peer-review does that, it throws out almost all of the spam; it pushes useful work of lower quality/impact down to less selective venues (generally with more specific subfields, so that I can read impactful work of domain X and less impactful work of subdomain X.2 but not X.1 and X.3); and even for accepted papers the process often results in significant improvements to them as a result of the reviews. Explicit fraud isn't that common, since it does have risks to the submitter that far outweigh the (limited) benefits of getting an extra publication on your CV.
• blastforses 8y ago
  The whole point of a peer review system is to detect fraud. If these bullshit papers got published, who knows how many of the other papers are of equal quality.
  • TheCoelacanth 8y ago
    The point of peer review is to stop bad research, not fake research. The reviewers are supposed to make sure that the methods used to acquire the data are clearly documented and that the conclusions drawn from the data follow from the data that was collected. They can't verify that the data collected actually was collected without literally watching over the shoulder of the researcher while the researcher was collecting it, which is obviously not feasible. A researcher actively trying to deceive the reviewer with fake data but with a good method for collecting that data and analysis of the data that would be correct if the data were real is not something that should be caught by a reviewer. That should be caught by someone reproducing the research after publication.
    • pervycreeper 8y ago
      >The point of peer review is to stop bad research, not fake research I would submit that "fake research" is, in fact, bad research.
      • UncleMeat 8y ago
        To a reviewer with three hours to spare, fake research is indistinguishable from good research. Unless you want to give me months to review a paper, I will not be able to detect fraud. Instead I rely on the fact that the community will attempt to integrate these ideas after publication and arrive at the conclusion that the paper was garbage.
        • malvosenior 8y ago
          If you only have three hours, can't detect fraud, can't tell if the the paper is even real. Then what's the point of even doing the peer review? Again, peer review offered no value at all in this case.