15 ms·

An Elaborate Academic-Journal Hoax

119 points by lukev 8y ago

lukev 8y ago
Representative quote: > "I am so utterly unimpressed," wrote Jacob T. Levy, a political theorist at McGill University, "by the fact that an enterprise that relies on a widespread presumption of not-fraud can be fooled some of the time by three people with Ph.D.s who spend 10 months deliberately trying to defraud it."
- throwaway5250 8y ago
  Representative quotes: "Is there any idea so outlandish that it won’t be published in a Critical/PoMo/Identity/‘Theory’ journal?" tweeted the Harvard psychologist Steven Pinker. "Three intrepid academics," wrote Yascha Mounk, an author and lecturer on government at Harvard, "just perpetrated a giant version of the Sokal Hoax, placing … fake papers in major academic journals. Call it Sokal Squared. The result is hilarious and delightful. It also showcases a serious problem with big parts of academia."
  - tptacek 8y ago
    Mounk didn't read the hoaxers writeup carefully enough to be a reliable judge; he gleefully tweeted two papers from the hoax that were rejected everywhere they were submitted, as evidence of the important problem the hoax uncovered --- this despite the fact that the hoaxers did in fact provide an accounting of what got accepted where (even though their writeup at the beginning of the piece misrepresented that accounting). From what Pinker wrote, who knows if he even read it?
- debacle 8y ago
  Shouldn't the default presupposition of any research be a lack of novelty?
  - Trombone12 8y ago
    I think generally people tend to assume that colleagues aren't dishonest, and mostly peer-review is supposed to catch honest mistakes. You could try working without any faith in your colleagues of course, but you are probably better of switching workplace if that sounds like reasonable behaviour to you.
    - debacle 8y ago
      What your describing sounds nothing like rigor.
      - UncleMeat 8y ago
        The rigor comes from the experiment and the future analysis by other researchers, not from the review. You are expecting something from peer review that never exists.
- the8472 8y ago
  > can be fooled some of the time 7⁄20 or 7⁄17, depending on how you want to count is concerning.
- malvosenior 8y ago
  Alternative take: A system designed to detect bad research failed to detect blatant falsehoods, algorithmically generated noise, rabid propaganda and other seemingly easy things to detect. That PhDs worked on this doesn’t mean the system didn’t fail spectacularly.
  - ModernMech 8y ago
    > doesn’t mean the system didn’t fail spectacularly. You're mischaracterizing the system; it doesn't end with publication. After publication, the science goes out to a wider audience of scientists who can build on the paper or refute it if they find it's wrong. Sometimes this process can take a long time and many cycles of publicans, but that's just the nature of science. We have to stop pretending like the peer review process is some kind of magical filter of true science, or that it was ever intended to be as such. That notion leads to the faulty idea that if it's published, it's true. Junk science gets published, even if it's not disingenuous, and that's probably true no matter how great you make the peer review system.
    - malvosenior 8y ago
      I'm not saying that peer review is the end all be all of truth, but it does serve some purpose. And in this case it didn't provide any value at all.
    - pochamago 8y ago
      What is the point of journals if quite literally anything can be published in them? Isn't the point that they serve as a basic filter for obvious noise? Shouldn't we be critical of a peer review process that doesn't function at even the most basic level?
      - PeterisP 8y ago
        A filter that does filter out the vast majority of obvious noise is very useful even if it lets something through. And the current peer-review does that, it throws out almost all of the spam; it pushes useful work of lower quality/impact down to less selective venues (generally with more specific subfields, so that I can read impactful work of domain X and less impactful work of subdomain X.2 but not X.1 and X.3); and even for accepted papers the process often results in significant improvements to them as a result of the reviews. Explicit fraud isn't that common, since it does have risks to the submitter that far outweigh the (limited) benefits of getting an extra publication on your CV.
- blastforses 8y ago
  The whole point of a peer review system is to detect fraud. If these bullshit papers got published, who knows how many of the other papers are of equal quality.
  - TheCoelacanth 8y ago
    The point of peer review is to stop bad research, not fake research. The reviewers are supposed to make sure that the methods used to acquire the data are clearly documented and that the conclusions drawn from the data follow from the data that was collected. They can't verify that the data collected actually was collected without literally watching over the shoulder of the researcher while the researcher was collecting it, which is obviously not feasible. A researcher actively trying to deceive the reviewer with fake data but with a good method for collecting that data and analysis of the data that would be correct if the data were real is not something that should be caught by a reviewer. That should be caught by someone reproducing the research after publication.
    - pervycreeper 8y ago
      >The point of peer review is to stop bad research, not fake research I would submit that "fake research" is, in fact, bad research.
      - UncleMeat 8y ago
        To a reviewer with three hours to spare, fake research is indistinguishable from good research. Unless you want to give me months to review a paper, I will not be able to detect fraud. Instead I rely on the fact that the community will attempt to integrate these ideas after publication and arrive at the conclusion that the paper was garbage.
        malvosenior 8y ago
        If you only have three hours, can't detect fraud, can't tell if the the paper is even real. Then what's the point of even doing the peer review? Again, peer review offered no value at all in this case.
tinalumfoil 8y ago
I've been more disturbed by the response to this than the papers themselves. Saying,well of course we were duped, you really expected the peer review system to not to utterly fail in the presence of a few bad actors? does not inspire my faith in the system.
- eindiran 8y ago
  That's what stuck out to me as well. This quote: "I am so utterly unimpressed," wrote Jacob T. Levy, a political theorist at McGill University, "by the fact that an enterprise that relies on a widespread presumption of not-fraud can be fooled some of the time by three people with Ph.D.s who spend 10 months deliberately trying to defraud it." Isn't the ability to separate out the quality work from the garbage the point of the peer review system? If the system was working as intended, the reviewers even with the assumption that they aren't being defrauded should have been able to sort the papers into the garbage pile.
  - davemp 8y ago
    > an enterprise that relies on a widespread presumption of not-fraud This is section really gets me. How in the world is science supposed to rely on “not-fraud”? Sure if you fake your data you might dupe a diligent reviewer, but this is why results are supposed to be reproduced. Anything that relies on “not-fraud” seems inherently unscientific and straight up naive.
    - BurritoAlPastor 8y ago
      Social sciences have pretty deep-seated logistical obstacles to reproducibility. If you want to reproduce an archaeology paper, well, how? You can’t exactly go dig the same site back up. If you go out as a cultural anthropologist to hang out with some indigenous tribe somewhere, and your experiences aren’t the same as the last guy, does that mean anything? How much is because you’re different people? How much is because the tribe has already gotten used to having a researcher around? This is part of why shit like the Piltdown Man happens.
    - 8y ago
debacle 8y ago
This wasn't a hoax it was social commentary and an experiment in prejudice. That nazi language can be published under a progressive guise should be chilling to anyone who is a vehicle for these ideals.
- ArchTypical 8y ago
  It wasn't a hoax. It's a confounding detail that a meta-commentary article is employing.
  - AnimalMuppet 8y ago
    Sorry, I can't parse that at all. Could you be a bit clearer on what you're trying to say here?
    - ArchTypical 8y ago
      The article is attempting to explain the events, by characterizing them as a type of deception when it shows the exact opposite (it's not a hoax that papers are accepted on political aim, rather than evidence). It's a small detail, but telling as to the intent and voracity without knowing most the story.
throwaway5250 8y ago
If the people at the top of your discipline can't tell the difference between quality and inane garbage, what do you really have?
- AnimalMuppet 8y ago
  Not a discipline.
- deleted 8y ago
  [deleted]
panarky 8y ago
It's not just the humanities, computer science is also susceptible to hoaxes built from jargon. SCIgen works like an academic "Mad Libs" of sorts, arbitrarily slotting in computer-science buzzwords like "distributed hash tables" and "Byzantine fault tolerance." http://news.mit.edu/2015/how-three-mit-students-fooled-scientific-journals-0414 http://news.mit.edu/2015/how-three-mit-students-fooled-scien...
- ajshankar 8y ago
  That paper was accepted as a non-reviewed paper. There was no peer-review process.
  - Apocryphon 8y ago
    Which one? Both that article and this one lists multiple cases where SCIgen-generated papers were waved through: https://en.wikipedia.org/wiki/SCIgen#Prominent_results https://en.wikipedia.org/wiki/SCIgen#Prominent_results
    - ajshankar 8y ago
      From the article: In April of 2005 the team’s submission, “Rooter: A Methodology for the Typical Unification of Access Points and Redundancy,” was accepted as a non-reviewed paper to the World Multiconference on Systemics, Cybernetics and Informatics (WMSCI), a conference that Krohn says is known for “being spammy and having loose standards.”
currymj 8y ago
Can someone comment on whether these are respectable journals or not, within their fields? Every few years someone will do a Sokal-hoax-esque thing in scientific fields, but usually they're just getting a paper into very-low-quality predatory journals that have only a nominal peer review process. I'm wondering if that's what happened here. Do humanities fields even have the same set of incentives leading to a proliferation of those sorts of journals?
- mattnewport 8y ago
  "We set out with three basic rules: (1) we’ll focus almost exclusively upon ranked peer-reviewed journals in the field, the higher the better and at the top of their subdisciplines whenever possible; (2) we will not pay to publish any paper; and (3) if we are asked at any point by a journal editor or reviewer (but not a journalist!) if any paper we wrote is an attempted hoax, we will admit it. These rules were meant to ensure that any conclusions we derived from the field came from the field itself, not the unrelated but significant problem that also corrupts academic pursuits: the proliferation of predatory and quasi-predatory journals with extremely low standards. With these rules guiding us, we committed to transparently reporting the results, whether we succeeded or failed." https://areomagazine.com/2018/10/02/academic-grievance-studies-and-the-corruption-of-scholarship/ https://areomagazine.com/2018/10/02/academic-grievance-studi... I'm not qualified to say if these truly were considered respectable journals in their fields but the authors had that as an explicit aim.
- Trombone12 8y ago
  Well, they claim to have targeted "top journals" in the field, but it's not like "intersectional gender studies" is a particularly big field. I looked up two journals they had published in and their articles got less than one citation per year typically ([1]&[2]), not exactly a booming field. [1]: https://www.scimagojr.com/journalsearch.php?q=145138&tip=sid&clean=0 https://www.scimagojr.com/journalsearch.php?q=145138&tip=sid... [2]: https://www.scimagojr.com/journalsearch.php?q=14798&tip=sid&clean=0 https://www.scimagojr.com/journalsearch.php?q=14798&tip=sid&...
  - emmelaich 8y ago
ahelwer 8y ago
Most published stuff is irrelevant crap that will never be cited or included in the central canon. This is true even in computer science. Also, as we know from computer security, system design becomes massively more difficult if it requires hardening against malicious actors. The presumption of good faith is a valuable component of any community that shouldn't be abused to score stupid points. For these reasons, Sokal Hoaxes (and their like) are damaging wastes of time and the perpetrators should not be praised.
- hnaccy 8y ago
  >The presumption of good faith is a valuable component of any community that shouldn't be abused to score stupid points. Shouldn't we be concerned with the possibility of undetected bad actors who are abusing it for more nefarious reasons?
msabalau 8y ago
Given that is an experiment on human subjects, involving deception, the IRB review should be interesting. Or weren't they aren't concerned about that type of ethics?
- gwern 8y ago
  What IRB approved the existence of these journals? Or weren't they concerned about that type of ethics?
theseatoms 8y ago
All of the above, because it exposes vulnerability to bad faith.
haberman 8y ago
This isn't simply a case of poor quality control. Sure, hoax articles have made it through some (largely pay-to-publish) journals in other fields from time to time. And yes those incidents should inspire reflection and correction, as well as a hit to the reputations of such journals. But this is much more. This is about a set of ideas that pass for expert scholarship despite having lost any tethering to to the real world. Just look at this reaction to the hoax: > Some of the pieces were indeed outlandish (and not all were accepted). But it appears that some were simply based in premises (e.g. social constructionism) or political principles (e.g trans equality) that the hoax authors find problematic but we do not. https://twitter.com/alisonphipps/status/1047421076509261824 https://twitter.com/alisonphipps/status/1047421076509261824 Note that word "problematic." I don't want to know whether social constructionism is considered "problematic", I want to know whether it is true. There is an actual real world out there that scholars are purporting to study. But when science and empiricism are considered oppressive, all we are left with is posturing and grandstanding over ideology.
- lukev 8y ago
  If your criteria for publication is what is epistemologically true you are going to find it difficult to publish in any domain except possibly pure mathematics or physics.
  - haberman 8y ago
    Social Constructionism and Blank Slatism are theories that can be tested. There is plenty that can be published on these topics that is based in evidence rather than ideology.
    - PeterisP 8y ago
      However, if you fake (or, as in this case, completely make up) your evidence, then you'll be able to get false things published - the peer review process has no ability to verify if what you claim is true, it does not and will not re-test the theory, that might be done by other teams after your results are published but not before.
deleted 8y ago
[deleted]
neonate 8y ago
The WSJ editorial linked to has more info: http://archive.is/llwSP http://archive.is/llwSP.
Sniffnoy 8y ago
> Karl Steel, an associate professor of English at Brooklyn College and the Graduate Center of the City University of New York, called the trio’s work "simply not rigorous research" and described three objections to it. It is too narrow in disciplinary scope, he said. It focuses on exposing weaknesses in gender and ethnic studies, conspicuously ideological fields, when that effort would be better spent looking at more-substantive problems like the replication crisis in psychology, or unfounded scholarly claims in cold fusion or laissez-faire economics. Yeah... that's the point! The whole point here is that you have these "conspicuously ideological" fields or subfields, which are not doing real scholarship, not interested in truth, but which rather are pushing a specific position and attempting to label as evil and shout down anyone who disagrees; and yet they're able to get away with claiming that they're doing real work, that this is what real scholarship is, and it's seriously damaging, especially (as the hoaxers note) to anyone who does want to do real truth-seeking work in these areas. Sure, to those of us who realize what's going on, more hardly needs to be said. But lots of people don't, and the "grievance scholars" are slowly convincing the rest of academia that real scholarship consists not of neutral methods and honest argument, but agreeing with the grievance scholars in all things, and finding ways to inject it whether it's relevant or not. The point of this hoax is to make obvious what already should be. To put it another way -- "grievance studies" is, as Dr. Steel says, an easy epistemological target. But it's a hard social target. It's built up a layer of protection around itself consisting not of truth but of social pressure and demands to conform or be ostracized. No claim, true or false, should be so protected, and that shield needs to be shattered. It's only an "easy target" in the imaginary world where people are rational and such transparent bullshit doesn't work. But in reality, it does, which is why there's a need to puncture it.
- harryh 8y ago
  Also "not rigorous research" != "effort would be better spent looking at more-substantive problems." Rigor and importance are orthogonal concepts. Someone could do research of great rigor on a completely unimportant problem and people do bad research all the time on important ones.
- tomlock 8y ago
  > and the "grievance scholars" are slowly convincing the rest of academia that real scholarship consists not of neutral methods and honest argument, but agreeing with the grievance scholars in all things, and finding ways to inject it whether it's relevant or not. What degrees or disciplines would you characterise as "grievance scholars", and what evidence do you have that suggests the influence those disciplines have over scholarship is increasing? The amount of degrees awarded in women's studies over the last 10 years has significantly dropped.
Tade0 8y ago
The trio could have reached out to colleagues in physics and other fields, but instead opted for "poor experimental design." They could, but the Bogdanov brothers did just that years earlier.
entwife 8y ago
Journals are for reports from scientists and academics. Would they be more useful if there was some additional review of the authors reputation, and the reliability of the particular report? Upvoting or public signing of the documents, or public signed comments. An expert in a field is more familiar with the reputation of their colleagues than outsiders.
brianberns 8y ago
> I don't want to know whether social constructionism is considered "problematic", I want to know whether it is true. Not all legitimate scholarly work is based on a search for scientific truth. E.g. Interpretations/analysis of Shakespeare's work.
edouard-harris 8y ago
The authors state that this was a long term research project cut short. Which suggests the question: If you played this game long enough, undetected, could you get real scholars to cite your fake research? Future work, perhaps. (I don't have any direct evidence of this, but I strongly suspect this is an hypothesis that the authors would have wanted to test.)
- PinkMilkshake 8y ago
  I hope there is someone out there right now, working towards a degree in Critical <whatever> Studies for no other reason than to get a PhD publishing nonsense and reveal it all afterwards.
PinkMilkshake 8y ago
I cant help but give a Gilfoyle style heh... heh... heh.... What I hope has happened is the 20 revealed papers are not actually all of them and that they do a Wikileaks style drip release, wait on a response, then hit us with more much later.
- amw 8y ago
  If they had those results, they would have put them front and center. Just like if the TSA had ever caught a terrorist, it would be on the 24-hour networks for days.
tptacek 8y ago
Discussed earlier on HN: https://news.ycombinator.com/item?id=18127811 https://news.ycombinator.com/item?id=18127811 The hoaxers writeup is, I think, pretty misleading. Their work tended to get rejected from high-impact journals. They submitted multiple stories to some journals (notably, to Hypatia, which they brag about getting accepted in) and leave a clear impression that their most ridiculous work was what made it in, rather than a much more banal paper. They quote almost entirely favorable feedback --- including, in their earlier writeup, from papers that were ultimately rejected! They brag about papers that were ultimately rejected. They submit approximately the same paper to multiple journals and are rejected by all but the very most marginal. In at least one case, they cite the same reviewer multiple times, leaving the impression of a consensus. If you don't think that's misleading, it's worth knowing that Yascha Mounk, cited warmly in this piece, was clearly taken in, dazzled on Twitter by a rejected paper from the hoax. What have they really uncovered? 1. Peer review isn't replication and never could have been. Peer reviewers get, at best, a couple hours, unpaid; submitting researchers get many months, and are often grant-funded. The hoaxers took ten months to get a few papers accepted in marginal journals. The pop understanding of what peer review is is, simply, broken: peer review depends on good faith, and it was never realistic to believe otherwise. 2. There are lots of crappy journals. Didn't you already know that? The market responds to the demand that professional academics publish, by creating venues that ensure professional academics can reliably publish. That's why we have measures like "impact factors". This is not at all a phenomenon isolated to the social sciences. If the authors even acknowledged these conclusions, they'd take all the excitement out of their hoax. We'd all nod our heads and say, "of course". Instead, they pretend to have taken down something they've dubbed "grievance studies", while pretending that they couldn't have done the same thing to, say, a marginal economics journal.
- fenomas 8y ago
  It's hard to tell specifically how their methodology went, but this bit of the author's write-up jumps out: > As for our performance, 80% of our papers overall went to full peer review, which keeps with the standard 10-20% of papers that are “desk rejected” without review at major journals across the field. We improved this ratio from 0% at first to 94.4% after a few months of experimenting with much more hoaxish papers. This makes it sound like the authors wrote a bunch of absurd articles, all of which got desk rejected, and then they iterated until they got papers accepted somewhere. With that methodology, the conclusion ("we got papers accepted somewhere!") doesn't seem very surprising. Edited to add: if the above is correct and their early submissions were all rejected, one could reasonably say of the results: "Editorial standards at all journals surveyed were more rigorous than the authors initially expected".
smadge 8y ago
If the author’s intention was to show the dubious quality of research in these fields, wouldn’t a better approach have been to select high impact articles from prestigious journals in the fields which they believed exemplified the problem, and then written rebuttals/critiques arguing why they are flawed?
- rule_follower 8y ago
  If the particular research is non-falsifiable and the critic is convinced the entire field is not an academic discipline at all but a cult with its entirely own internal logic with no relation to reality, how would an objective rebuttal be constructed? Is the right way for atheists to engage with religious scholars to critique the most nonsensical output of theologians?
jancsika 8y ago
> To answer these questions, this article engages feminist geography and broader feminist literature and draws on nearly 1000 h of public observations of dogs and their human companions conducted at three dog parks in Southeast Portland, Oregon, beginning on 10 June 2016, and ending on 10 June 2017. In Sokal's single hoax in 1996 he (and his buddies, I guess) made a claim, followed it by a non-sequitur of true but uninteresting sentences about physics, then repeated the claim. It was a sandwich with no meat in it, and the (small, non-peer-reviewed) post-modern journal ate it up. Sokal wanted to show that a certain style of post-modern writing substituted scientific jargon for an argument (or at least it used references to esoteric scientific fields as pretentious, uninteresting metaphors). The elegance of parodying this style was that his submission contained no data at all. It wasn't merely that the conclusion of the submission wouldn't be replicated. It was that any conscious human should notice the complete lack of supporting evidence for the claim! The upshot of the journal accepting the article was that "the emperor had no clothes," so to speak. Sokal went on to write a book showing how several prominent post-modern thinkers used scientific non-sequiturs to make their non-scientific ideas sound more weighty. (Just as he had in his hoax.) These three authors, however, are going a different route from Sokal-- specifically, one that provides fake data as evidence of a claim. (Or I guess they could have actually sat at dog parks for 1000 hours, but it doesn't matter for the point I'm making.) By providing fake data, this hoax passes an initial filter which the Sokal hoax did not-- namely, the filter that should reject submissions which lack any supporting evidence whatsoever. That means this hoax only reveals a lack of replication and/or verification, whereas the Sokal hoax reveals a lack of both peer review and basic reading comprehension even by a non-physicist. That makes this a weaker class of hoax than the Sokal hoax. Finally, the authors purport to be going after an entire "ideological slant" in academia. Whatever one thinks of that aim, it is enormous and amorphous when compared to the narrow aim of Sokal keeping scientific jargon from being misused/misplaced. So to recap: * 1996: strong hoax successful against a single pomo journal, weak but persuasive claim against coopting scientific jargon for non-scientific aims * 2018: weak hoax successful against 7 of (total number?) journals, strong and unpersuasive claim against all leftist academic fields Now I'm curious-- did anyone in the late 90s criticize Sokal's hoax on the grounds that his careful constraints would be ignored by future pranksters? Edit: clarification
newintellectual 8y ago
Those criticizing the hoaxers are missing their central point: a real field of study wouldn't be taken in by stuff that outsiders with any hint of common sense would very swiftly recognize as laughable nonsense and garbage. The real horror story is that there are so-called fields that are nothing except laughable nonsense garbage. When you call rotting garbage food, don't be surprised when rotting garbage contributed by outsiders is impossible to be differentiated from your official garbage.
outsideacademe 8y ago
[throw away account as I am a recovering academic and wouldn't want to impact colleagues who have not been able to escape] Everyone is distracted by the politics here when the real story is the effort. Three people layman to these fields achieved a near tenure track publication list/ratio in a little over a year. I have a STEM PhD and I know that me and three of my friends couldn't do the same if we tried in a closely related STEM discipline much less an entirely different field. Hypatia is the journal of record for feminist philosophy (and one of the few respected feminist journals in general). It should not be possible to go from 0 to published in a "real" top tier journal. If the paper is "not too crazy" that's even more damning! They produced a "real" paper as a joke? Do you really think that these same authors could get a paper published in The International Journal of Computer Vision in a little less than a year even if they were using fraudulent data? It would take them two years to know what they could and couldn't fake without being caught. If the liberal arts don't want the sneering dismissal of their fields as low effort to become culturally entrenched they need to dump or reform the "X studies" departments and journals.
int_19h 8y ago
I see a lot of people complaining that it is done in bad faith, and doesn't prove much. But is there really much to prove? What's the baseline? I mean, we're talking about a field where this got published - and it was not a hoax: http://journals.sagepub.com/doi/full/10.1177/0309132515623368 http://journals.sagepub.com/doi/full/10.1177/030913251562336... The people behind this hoax had an unenviable task of matching and beating a paper like that (since, apparently, by itself it was not sufficient to prove the point). How would you do so?
- foldr 8y ago
  Bad papers get published in all fields. Linking to one bad journal article shows nothing. Here, for example, is a reference to a bad paper published in a field which is not usually the target of this kind of criticism: https://academia.stackexchange.com/questions/9602/rediscovery-of-calculus-in-1994-what-should-have-happened-to-that-paper https://academia.stackexchange.com/questions/9602/rediscover... https://math.berkeley.edu/~ehallman/math1B/TaisMethod.pdf https://math.berkeley.edu/~ehallman/math1B/TaisMethod.pdf
  - int_19h 8y ago
    True, but when such papers are discovered, they're usually torn down by people in the field. In this case, it was the other way around - all meaningful criticism came from the outside, and in response, researchers in the field mostly circled the wagons. Besides, the paper in question wasn't merely published - the author got a prestigious NSF award that lists that paper: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1253779 https://www.nsf.gov/awardsearch/showAward?AWD_ID=1253779 This, to me, implies that either such papers are considered broadly acceptable in the field, or they're not actually read. Not sure which is worse.
    - foldr 8y ago
      Have you read the paper you linked to, or just the abstract and quotes in the right wing press? You're confused about how NSF grants work. The publication is listed as an output. The grant was awarded before that paper was published. The listing of the paper on that page implies nothing about its quality. My example paper was torn down by mathematicians, not diabetes researchers.
seany 8y ago
This is the best thing I've seen in quite a while.
pervycreeper 8y ago
I've noticed a lot of hair-splitting in these comments over whether this was a failure of peer review, whether to call the papers, considered as specimens of scholarship in their appropriate fields, "fraudulent" rather than "bad", or whether the number of hours they actually logged at the dog park is a relevant concern. All of this is quite besides the point. The researchers passed a version of the Turing Test among the reviewers of their papers. All that we need from that point is an acknowledgement of the absurdity and falseness of the papers themselves. This may not be easily forthcoming, though. One of the reviewers of Social Text took the position, in the wake of Sokal's hoax, that his paper constituted good scholarship, despite the stated intentions of its author. Also confer with Poe's Law. However, when we admit this as a possible response, we are in the territory of radical relativism, with no way to adjudicate between the claims of these scholars and their critics, and for that matter, those of religious fanatics, mentally ill delusional people, confidence scammers, or indeed anybody at all. This is a position which seems, at least, to be somewhat at odds with the fundamental scholarly enterprise. At its heart, one has to cross the pons arsinorum of admitting that, yes, the papers are indeed absurd nonsense. At this point, it seems impossible to convince some people to take this step. If the critics's objections could be distilled into something with a semblance of validity and germaneness, then it would be that the researchers's methods were too unfocused. But had they taken a narrower approach, the response could just have been to rationalize and ignore. Fight fire with fire (being generality, in this case).

119 points by lukev 8y ago

Representative quote: > "I am so utterly unimpressed," wrote Jacob T. Levy, a political theorist at McGill University, "by the fact that an enterprise that relies on a widespread presumption of not-fraud can be fooled some of the time by three people with Ph.D.s who spend 10 months deliberately trying to defraud it."

throwaway5250 8y ago
Representative quotes: "Is there any idea so outlandish that it won’t be published in a Critical/PoMo/Identity/‘Theory’ journal?" tweeted the Harvard psychologist Steven Pinker. "Three intrepid academics," wrote Yascha Mounk, an author and lecturer on government at Harvard, "just perpetrated a giant version of the Sokal Hoax, placing … fake papers in major academic journals. Call it Sokal Squared. The result is hilarious and delightful. It also showcases a serious problem with big parts of academia."
- tptacek 8y ago
  Mounk didn't read the hoaxers writeup carefully enough to be a reliable judge; he gleefully tweeted two papers from the hoax that were rejected everywhere they were submitted, as evidence of the important problem the hoax uncovered --- this despite the fact that the hoaxers did in fact provide an accounting of what got accepted where (even though their writeup at the beginning of the piece misrepresented that accounting). From what Pinker wrote, who knows if he even read it?
debacle 8y ago
Shouldn't the default presupposition of any research be a lack of novelty?
- Trombone12 8y ago
  I think generally people tend to assume that colleagues aren't dishonest, and mostly peer-review is supposed to catch honest mistakes. You could try working without any faith in your colleagues of course, but you are probably better of switching workplace if that sounds like reasonable behaviour to you.
  - debacle 8y ago
    What your describing sounds nothing like rigor.
    - UncleMeat 8y ago
      The rigor comes from the experiment and the future analysis by other researchers, not from the review. You are expecting something from peer review that never exists.
the8472 8y ago
> can be fooled some of the time 7⁄20 or 7⁄17, depending on how you want to count is concerning.
malvosenior 8y ago
Alternative take: A system designed to detect bad research failed to detect blatant falsehoods, algorithmically generated noise, rabid propaganda and other seemingly easy things to detect. That PhDs worked on this doesn’t mean the system didn’t fail spectacularly.
- ModernMech 8y ago
  > doesn’t mean the system didn’t fail spectacularly. You're mischaracterizing the system; it doesn't end with publication. After publication, the science goes out to a wider audience of scientists who can build on the paper or refute it if they find it's wrong. Sometimes this process can take a long time and many cycles of publicans, but that's just the nature of science. We have to stop pretending like the peer review process is some kind of magical filter of true science, or that it was ever intended to be as such. That notion leads to the faulty idea that if it's published, it's true. Junk science gets published, even if it's not disingenuous, and that's probably true no matter how great you make the peer review system.
  - malvosenior 8y ago
    I'm not saying that peer review is the end all be all of truth, but it does serve some purpose. And in this case it didn't provide any value at all.
  - pochamago 8y ago
    What is the point of journals if quite literally anything can be published in them? Isn't the point that they serve as a basic filter for obvious noise? Shouldn't we be critical of a peer review process that doesn't function at even the most basic level?
    - PeterisP 8y ago
      A filter that does filter out the vast majority of obvious noise is very useful even if it lets something through. And the current peer-review does that, it throws out almost all of the spam; it pushes useful work of lower quality/impact down to less selective venues (generally with more specific subfields, so that I can read impactful work of domain X and less impactful work of subdomain X.2 but not X.1 and X.3); and even for accepted papers the process often results in significant improvements to them as a result of the reviews. Explicit fraud isn't that common, since it does have risks to the submitter that far outweigh the (limited) benefits of getting an extra publication on your CV.
blastforses 8y ago
The whole point of a peer review system is to detect fraud. If these bullshit papers got published, who knows how many of the other papers are of equal quality.
- TheCoelacanth 8y ago
  The point of peer review is to stop bad research, not fake research. The reviewers are supposed to make sure that the methods used to acquire the data are clearly documented and that the conclusions drawn from the data follow from the data that was collected. They can't verify that the data collected actually was collected without literally watching over the shoulder of the researcher while the researcher was collecting it, which is obviously not feasible. A researcher actively trying to deceive the reviewer with fake data but with a good method for collecting that data and analysis of the data that would be correct if the data were real is not something that should be caught by a reviewer. That should be caught by someone reproducing the research after publication.
  - pervycreeper 8y ago
    >The point of peer review is to stop bad research, not fake research I would submit that "fake research" is, in fact, bad research.
    - UncleMeat 8y ago
      To a reviewer with three hours to spare, fake research is indistinguishable from good research. Unless you want to give me months to review a paper, I will not be able to detect fraud. Instead I rely on the fact that the community will attempt to integrate these ideas after publication and arrive at the conclusion that the paper was garbage.
      - malvosenior 8y ago
        If you only have three hours, can't detect fraud, can't tell if the the paper is even real. Then what's the point of even doing the peer review? Again, peer review offered no value at all in this case.

tinalumfoil 8y ago

I've been more disturbed by the response to this than the papers themselves. Saying,well of course we were duped, you really expected the peer review system to not to utterly fail in the presence of a few bad actors? does not inspire my faith in the system.

eindiran 8y ago
That's what stuck out to me as well. This quote: "I am so utterly unimpressed," wrote Jacob T. Levy, a political theorist at McGill University, "by the fact that an enterprise that relies on a widespread presumption of not-fraud can be fooled some of the time by three people with Ph.D.s who spend 10 months deliberately trying to defraud it." Isn't the ability to separate out the quality work from the garbage the point of the peer review system? If the system was working as intended, the reviewers even with the assumption that they aren't being defrauded should have been able to sort the papers into the garbage pile.
- davemp 8y ago
  > an enterprise that relies on a widespread presumption of not-fraud This is section really gets me. How in the world is science supposed to rely on “not-fraud”? Sure if you fake your data you might dupe a diligent reviewer, but this is why results are supposed to be reproduced. Anything that relies on “not-fraud” seems inherently unscientific and straight up naive.
  - BurritoAlPastor 8y ago
    Social sciences have pretty deep-seated logistical obstacles to reproducibility. If you want to reproduce an archaeology paper, well, how? You can’t exactly go dig the same site back up. If you go out as a cultural anthropologist to hang out with some indigenous tribe somewhere, and your experiences aren’t the same as the last guy, does that mean anything? How much is because you’re different people? How much is because the tribe has already gotten used to having a researcher around? This is part of why shit like the Piltdown Man happens.
  - 8y ago

debacle 8y ago

This wasn't a hoax it was social commentary and an experiment in prejudice. That nazi language can be published under a progressive guise should be chilling to anyone who is a vehicle for these ideals.

ArchTypical 8y ago
It wasn't a hoax. It's a confounding detail that a meta-commentary article is employing.
- AnimalMuppet 8y ago
  Sorry, I can't parse that at all. Could you be a bit clearer on what you're trying to say here?
  - ArchTypical 8y ago
    The article is attempting to explain the events, by characterizing them as a type of deception when it shows the exact opposite (it's not a hoax that papers are accepted on political aim, rather than evidence). It's a small detail, but telling as to the intent and voracity without knowing most the story.

throwaway5250 8y ago

If the people at the top of your discipline can't tell the difference between quality and inane garbage, what do you really have?

AnimalMuppet 8y ago
Not a discipline.
deleted 8y ago
[deleted]

panarky 8y ago

It's not just the humanities, computer science is also susceptible to hoaxes built from jargon. SCIgen works like an academic "Mad Libs" of sorts, arbitrarily slotting in computer-science buzzwords like "distributed hash tables" and "Byzantine fault tolerance." http://news.mit.edu/2015/how-three-mit-students-fooled-scientific-journals-0414 http://news.mit.edu/2015/how-three-mit-students-fooled-scien...

ajshankar 8y ago
That paper was accepted as a non-reviewed paper. There was no peer-review process.
- Apocryphon 8y ago
  Which one? Both that article and this one lists multiple cases where SCIgen-generated papers were waved through: https://en.wikipedia.org/wiki/SCIgen#Prominent_results https://en.wikipedia.org/wiki/SCIgen#Prominent_results
  - ajshankar 8y ago
    From the article: In April of 2005 the team’s submission, “Rooter: A Methodology for the Typical Unification of Access Points and Redundancy,” was accepted as a non-reviewed paper to the World Multiconference on Systemics, Cybernetics and Informatics (WMSCI), a conference that Krohn says is known for “being spammy and having loose standards.”

currymj 8y ago

Can someone comment on whether these are respectable journals or not, within their fields? Every few years someone will do a Sokal-hoax-esque thing in scientific fields, but usually they're just getting a paper into very-low-quality predatory journals that have only a nominal peer review process. I'm wondering if that's what happened here. Do humanities fields even have the same set of incentives leading to a proliferation of those sorts of journals?

mattnewport 8y ago
"We set out with three basic rules: (1) we’ll focus almost exclusively upon ranked peer-reviewed journals in the field, the higher the better and at the top of their subdisciplines whenever possible; (2) we will not pay to publish any paper; and (3) if we are asked at any point by a journal editor or reviewer (but not a journalist!) if any paper we wrote is an attempted hoax, we will admit it. These rules were meant to ensure that any conclusions we derived from the field came from the field itself, not the unrelated but significant problem that also corrupts academic pursuits: the proliferation of predatory and quasi-predatory journals with extremely low standards. With these rules guiding us, we committed to transparently reporting the results, whether we succeeded or failed." https://areomagazine.com/2018/10/02/academic-grievance-studies-and-the-corruption-of-scholarship/ https://areomagazine.com/2018/10/02/academic-grievance-studi... I'm not qualified to say if these truly were considered respectable journals in their fields but the authors had that as an explicit aim.
Trombone12 8y ago
Well, they claim to have targeted "top journals" in the field, but it's not like "intersectional gender studies" is a particularly big field. I looked up two journals they had published in and their articles got less than one citation per year typically ([1]&[2]), not exactly a booming field. [1]: https://www.scimagojr.com/journalsearch.php?q=145138&tip=sid&clean=0 https://www.scimagojr.com/journalsearch.php?q=145138&tip=sid... [2]: https://www.scimagojr.com/journalsearch.php?q=14798&tip=sid&clean=0 https://www.scimagojr.com/journalsearch.php?q=14798&tip=sid&...
- emmelaich 8y ago

ahelwer 8y ago

Most published stuff is irrelevant crap that will never be cited or included in the central canon. This is true even in computer science. Also, as we know from computer security, system design becomes massively more difficult if it requires hardening against malicious actors. The presumption of good faith is a valuable component of any community that shouldn't be abused to score stupid points. For these reasons, Sokal Hoaxes (and their like) are damaging wastes of time and the perpetrators should not be praised.

hnaccy 8y ago
>The presumption of good faith is a valuable component of any community that shouldn't be abused to score stupid points. Shouldn't we be concerned with the possibility of undetected bad actors who are abusing it for more nefarious reasons?

msabalau 8y ago

Given that is an experiment on human subjects, involving deception, the IRB review should be interesting. Or weren't they aren't concerned about that type of ethics?

gwern 8y ago
What IRB approved the existence of these journals? Or weren't they concerned about that type of ethics?

theseatoms 8y ago

All of the above, because it exposes vulnerability to bad faith.

haberman 8y ago

This isn't simply a case of poor quality control. Sure, hoax articles have made it through some (largely pay-to-publish) journals in other fields from time to time. And yes those incidents should inspire reflection and correction, as well as a hit to the reputations of such journals. But this is much more. This is about a set of ideas that pass for expert scholarship despite having lost any tethering to to the real world. Just look at this reaction to the hoax: > Some of the pieces were indeed outlandish (and not all were accepted). But it appears that some were simply based in premises (e.g. social constructionism) or political principles (e.g trans equality) that the hoax authors find problematic but we do not. https://twitter.com/alisonphipps/status/1047421076509261824 https://twitter.com/alisonphipps/status/1047421076509261824 Note that word "problematic." I don't want to know whether social constructionism is considered "problematic", I want to know whether it is true. There is an actual real world out there that scholars are purporting to study. But when science and empiricism are considered oppressive, all we are left with is posturing and grandstanding over ideology.

lukev 8y ago
If your criteria for publication is what is epistemologically true you are going to find it difficult to publish in any domain except possibly pure mathematics or physics.
- haberman 8y ago
  Social Constructionism and Blank Slatism are theories that can be tested. There is plenty that can be published on these topics that is based in evidence rather than ideology.
  - PeterisP 8y ago
    However, if you fake (or, as in this case, completely make up) your evidence, then you'll be able to get false things published - the peer review process has no ability to verify if what you claim is true, it does not and will not re-test the theory, that might be done by other teams after your results are published but not before.

deleted 8y ago

[deleted]

neonate 8y ago

The WSJ editorial linked to has more info: http://archive.is/llwSP http://archive.is/llwSP.

Sniffnoy 8y ago

> Karl Steel, an associate professor of English at Brooklyn College and the Graduate Center of the City University of New York, called the trio’s work "simply not rigorous research" and described three objections to it. It is too narrow in disciplinary scope, he said. It focuses on exposing weaknesses in gender and ethnic studies, conspicuously ideological fields, when that effort would be better spent looking at more-substantive problems like the replication crisis in psychology, or unfounded scholarly claims in cold fusion or laissez-faire economics. Yeah... that's the point! The whole point here is that you have these "conspicuously ideological" fields or subfields, which are not doing real scholarship, not interested in truth, but which rather are pushing a specific position and attempting to label as evil and shout down anyone who disagrees; and yet they're able to get away with claiming that they're doing real work, that this is what real scholarship is, and it's seriously damaging, especially (as the hoaxers note) to anyone who does want to do real truth-seeking work in these areas. Sure, to those of us who realize what's going on, more hardly needs to be said. But lots of people don't, and the "grievance scholars" are slowly convincing the rest of academia that real scholarship consists not of neutral methods and honest argument, but agreeing with the grievance scholars in all things, and finding ways to inject it whether it's relevant or not. The point of this hoax is to make obvious what already should be. To put it another way -- "grievance studies" is, as Dr. Steel says, an easy epistemological target. But it's a hard social target. It's built up a layer of protection around itself consisting not of truth but of social pressure and demands to conform or be ostracized. No claim, true or false, should be so protected, and that shield needs to be shattered. It's only an "easy target" in the imaginary world where people are rational and such transparent bullshit doesn't work. But in reality, it does, which is why there's a need to puncture it.

harryh 8y ago
Also "not rigorous research" != "effort would be better spent looking at more-substantive problems." Rigor and importance are orthogonal concepts. Someone could do research of great rigor on a completely unimportant problem and people do bad research all the time on important ones.
tomlock 8y ago
> and the "grievance scholars" are slowly convincing the rest of academia that real scholarship consists not of neutral methods and honest argument, but agreeing with the grievance scholars in all things, and finding ways to inject it whether it's relevant or not. What degrees or disciplines would you characterise as "grievance scholars", and what evidence do you have that suggests the influence those disciplines have over scholarship is increasing? The amount of degrees awarded in women's studies over the last 10 years has significantly dropped.

Tade0 8y ago

The trio could have reached out to colleagues in physics and other fields, but instead opted for "poor experimental design." They could, but the Bogdanov brothers did just that years earlier.

entwife 8y ago

Journals are for reports from scientists and academics. Would they be more useful if there was some additional review of the authors reputation, and the reliability of the particular report? Upvoting or public signing of the documents, or public signed comments. An expert in a field is more familiar with the reputation of their colleagues than outsiders.

brianberns 8y ago

> I don't want to know whether social constructionism is considered "problematic", I want to know whether it is true. Not all legitimate scholarly work is based on a search for scientific truth. E.g. Interpretations/analysis of Shakespeare's work.

edouard-harris 8y ago

The authors state that this was a long term research project cut short. Which suggests the question: If you played this game long enough, undetected, could you get real scholars to cite your fake research? Future work, perhaps. (I don't have any direct evidence of this, but I strongly suspect this is an hypothesis that the authors would have wanted to test.)

PinkMilkshake 8y ago
I hope there is someone out there right now, working towards a degree in Critical <whatever> Studies for no other reason than to get a PhD publishing nonsense and reveal it all afterwards.

PinkMilkshake 8y ago

I cant help but give a Gilfoyle style heh... heh... heh.... What I hope has happened is the 20 revealed papers are not actually all of them and that they do a Wikileaks style drip release, wait on a response, then hit us with more much later.

amw 8y ago
If they had those results, they would have put them front and center. Just like if the TSA had ever caught a terrorist, it would be on the 24-hour networks for days.

tptacek 8y ago

Discussed earlier on HN: https://news.ycombinator.com/item?id=18127811 https://news.ycombinator.com/item?id=18127811 The hoaxers writeup is, I think, pretty misleading. Their work tended to get rejected from high-impact journals. They submitted multiple stories to some journals (notably, to Hypatia, which they brag about getting accepted in) and leave a clear impression that their most ridiculous work was what made it in, rather than a much more banal paper. They quote almost entirely favorable feedback --- including, in their earlier writeup, from papers that were ultimately rejected! They brag about papers that were ultimately rejected. They submit approximately the same paper to multiple journals and are rejected by all but the very most marginal. In at least one case, they cite the same reviewer multiple times, leaving the impression of a consensus. If you don't think that's misleading, it's worth knowing that Yascha Mounk, cited warmly in this piece, was clearly taken in, dazzled on Twitter by a rejected paper from the hoax. What have they really uncovered? 1. Peer review isn't replication and never could have been. Peer reviewers get, at best, a couple hours, unpaid; submitting researchers get many months, and are often grant-funded. The hoaxers took ten months to get a few papers accepted in marginal journals. The pop understanding of what peer review is is, simply, broken: peer review depends on good faith, and it was never realistic to believe otherwise. 2. There are lots of crappy journals. Didn't you already know that? The market responds to the demand that professional academics publish, by creating venues that ensure professional academics can reliably publish. That's why we have measures like "impact factors". This is not at all a phenomenon isolated to the social sciences. If the authors even acknowledged these conclusions, they'd take all the excitement out of their hoax. We'd all nod our heads and say, "of course". Instead, they pretend to have taken down something they've dubbed "grievance studies", while pretending that they couldn't have done the same thing to, say, a marginal economics journal.

fenomas 8y ago
It's hard to tell specifically how their methodology went, but this bit of the author's write-up jumps out: > As for our performance, 80% of our papers overall went to full peer review, which keeps with the standard 10-20% of papers that are “desk rejected” without review at major journals across the field. We improved this ratio from 0% at first to 94.4% after a few months of experimenting with much more hoaxish papers. This makes it sound like the authors wrote a bunch of absurd articles, all of which got desk rejected, and then they iterated until they got papers accepted somewhere. With that methodology, the conclusion ("we got papers accepted somewhere!") doesn't seem very surprising. Edited to add: if the above is correct and their early submissions were all rejected, one could reasonably say of the results: "Editorial standards at all journals surveyed were more rigorous than the authors initially expected".

smadge 8y ago

If the author’s intention was to show the dubious quality of research in these fields, wouldn’t a better approach have been to select high impact articles from prestigious journals in the fields which they believed exemplified the problem, and then written rebuttals/critiques arguing why they are flawed?

rule_follower 8y ago
If the particular research is non-falsifiable and the critic is convinced the entire field is not an academic discipline at all but a cult with its entirely own internal logic with no relation to reality, how would an objective rebuttal be constructed? Is the right way for atheists to engage with religious scholars to critique the most nonsensical output of theologians?

jancsika 8y ago

> To answer these questions, this article engages feminist geography and broader feminist literature and draws on nearly 1000 h of public observations of dogs and their human companions conducted at three dog parks in Southeast Portland, Oregon, beginning on 10 June 2016, and ending on 10 June 2017. In Sokal's single hoax in 1996 he (and his buddies, I guess) made a claim, followed it by a non-sequitur of true but uninteresting sentences about physics, then repeated the claim. It was a sandwich with no meat in it, and the (small, non-peer-reviewed) post-modern journal ate it up. Sokal wanted to show that a certain style of post-modern writing substituted scientific jargon for an argument (or at least it used references to esoteric scientific fields as pretentious, uninteresting metaphors). The elegance of parodying this style was that his submission contained no data at all. It wasn't merely that the conclusion of the submission wouldn't be replicated. It was that any conscious human should notice the complete lack of supporting evidence for the claim! The upshot of the journal accepting the article was that "the emperor had no clothes," so to speak. Sokal went on to write a book showing how several prominent post-modern thinkers used scientific non-sequiturs to make their non-scientific ideas sound more weighty. (Just as he had in his hoax.) These three authors, however, are going a different route from Sokal-- specifically, one that provides fake data as evidence of a claim. (Or I guess they could have actually sat at dog parks for 1000 hours, but it doesn't matter for the point I'm making.) By providing fake data, this hoax passes an initial filter which the Sokal hoax did not-- namely, the filter that should reject submissions which lack any supporting evidence whatsoever. That means this hoax only reveals a lack of replication and/or verification, whereas the Sokal hoax reveals a lack of both peer review and basic reading comprehension even by a non-physicist. That makes this a weaker class of hoax than the Sokal hoax. Finally, the authors purport to be going after an entire "ideological slant" in academia. Whatever one thinks of that aim, it is enormous and amorphous when compared to the narrow aim of Sokal keeping scientific jargon from being misused/misplaced. So to recap: * 1996: strong hoax successful against a single pomo journal, weak but persuasive claim against coopting scientific jargon for non-scientific aims * 2018: weak hoax successful against 7 of (total number?) journals, strong and unpersuasive claim against all leftist academic fields Now I'm curious-- did anyone in the late 90s criticize Sokal's hoax on the grounds that his careful constraints would be ignored by future pranksters? Edit: clarification

newintellectual 8y ago

Those criticizing the hoaxers are missing their central point: a real field of study wouldn't be taken in by stuff that outsiders with any hint of common sense would very swiftly recognize as laughable nonsense and garbage. The real horror story is that there are so-called fields that are nothing except laughable nonsense garbage. When you call rotting garbage food, don't be surprised when rotting garbage contributed by outsiders is impossible to be differentiated from your official garbage.

outsideacademe 8y ago

[throw away account as I am a recovering academic and wouldn't want to impact colleagues who have not been able to escape] Everyone is distracted by the politics here when the real story is the effort. Three people layman to these fields achieved a near tenure track publication list/ratio in a little over a year. I have a STEM PhD and I know that me and three of my friends couldn't do the same if we tried in a closely related STEM discipline much less an entirely different field. Hypatia is the journal of record for feminist philosophy (and one of the few respected feminist journals in general). It should not be possible to go from 0 to published in a "real" top tier journal. If the paper is "not too crazy" that's even more damning! They produced a "real" paper as a joke? Do you really think that these same authors could get a paper published in The International Journal of Computer Vision in a little less than a year even if they were using fraudulent data? It would take them two years to know what they could and couldn't fake without being caught. If the liberal arts don't want the sneering dismissal of their fields as low effort to become culturally entrenched they need to dump or reform the "X studies" departments and journals.

int_19h 8y ago

I see a lot of people complaining that it is done in bad faith, and doesn't prove much. But is there really much to prove? What's the baseline? I mean, we're talking about a field where this got published - and it was not a hoax: http://journals.sagepub.com/doi/full/10.1177/0309132515623368 http://journals.sagepub.com/doi/full/10.1177/030913251562336... The people behind this hoax had an unenviable task of matching and beating a paper like that (since, apparently, by itself it was not sufficient to prove the point). How would you do so?

foldr 8y ago
Bad papers get published in all fields. Linking to one bad journal article shows nothing. Here, for example, is a reference to a bad paper published in a field which is not usually the target of this kind of criticism: https://academia.stackexchange.com/questions/9602/rediscovery-of-calculus-in-1994-what-should-have-happened-to-that-paper https://academia.stackexchange.com/questions/9602/rediscover... https://math.berkeley.edu/~ehallman/math1B/TaisMethod.pdf https://math.berkeley.edu/~ehallman/math1B/TaisMethod.pdf
- int_19h 8y ago
  True, but when such papers are discovered, they're usually torn down by people in the field. In this case, it was the other way around - all meaningful criticism came from the outside, and in response, researchers in the field mostly circled the wagons. Besides, the paper in question wasn't merely published - the author got a prestigious NSF award that lists that paper: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1253779 https://www.nsf.gov/awardsearch/showAward?AWD_ID=1253779 This, to me, implies that either such papers are considered broadly acceptable in the field, or they're not actually read. Not sure which is worse.
  - foldr 8y ago
    Have you read the paper you linked to, or just the abstract and quotes in the right wing press? You're confused about how NSF grants work. The publication is listed as an output. The grant was awarded before that paper was published. The listing of the paper on that page implies nothing about its quality. My example paper was torn down by mathematicians, not diabetes researchers.

seany 8y ago

This is the best thing I've seen in quite a while.

pervycreeper 8y ago

I've noticed a lot of hair-splitting in these comments over whether this was a failure of peer review, whether to call the papers, considered as specimens of scholarship in their appropriate fields, "fraudulent" rather than "bad", or whether the number of hours they actually logged at the dog park is a relevant concern. All of this is quite besides the point. The researchers passed a version of the Turing Test among the reviewers of their papers. All that we need from that point is an acknowledgement of the absurdity and falseness of the papers themselves. This may not be easily forthcoming, though. One of the reviewers of Social Text took the position, in the wake of Sokal's hoax, that his paper constituted good scholarship, despite the stated intentions of its author. Also confer with Poe's Law. However, when we admit this as a possible response, we are in the territory of radical relativism, with no way to adjudicate between the claims of these scholars and their critics, and for that matter, those of religious fanatics, mentally ill delusional people, confidence scammers, or indeed anybody at all. This is a position which seems, at least, to be somewhat at odds with the fundamental scholarly enterprise. At its heart, one has to cross the pons arsinorum of admitting that, yes, the papers are indeed absurd nonsense. At this point, it seems impossible to convince some people to take this step. If the critics's objections could be distilled into something with a semblance of validity and germaneness, then it would be that the researchers's methods were too unfocused. But had they taken a narrower approach, the response could just have been to rationalize and ignore. Fight fire with fire (being generality, in this case).