3 ms·

I think it's surprising that on a site such as HN the narrative of these authors is so readily accepted without any second thought. The claim is that they just

by ATsch 8y ago

I think it's surprising that on a site such as HN the narrative of these authors is so readily accepted without any second thought. The claim is that they just made up trash and submitted it to high-quality peer-reviewed journals and they accepted them. The reality is much less outragous. Of the 20 papers, 4 were published and 3 more accepted for publication. They spent a year doing this. The authors note that they submitted the papers to "top journals in the field", leaving out the fact that all of those top journals rejected the 20 papers and they had to revise the papers and send them to shoddier journals instead. The review quoted by the authors was from an unsure grad student doing their first review of a paper. Said student later said they knew the paper was crap, but still tried to be constructive. The assertion of the authors is that this stuff getting published means the whole field is worthless. Which is of course ridiculous. Medicine didn't become obsolete when a journal published a paper with an abstract ending in "The fact that these last sentences appear in the published paper tell you, dear reader, exactly how seriously the editorial process has been taken". Computer Science didn't implode when three MIT students made an AI to generate random papers and 120(!) were accepted by the reputable IEEE and Springer. The blame was back then correctly put on the publishers and the general flaws of the current scientific process instread. So, as people have pointed out, it's ironic that their essay itself is incredibly scientifically unrigorous. No controls, unsourced claims, biased authors and conclusions that aren't supported by the data.

prepend 8y ago
“The assertion of the authors is that this stuff getting published means the whole field is worthless.” I don’t get this at all from the authors. My take is that they call out serious flaws in the rigor of how papers are reviewed. I don’t think it’s reasonable to extend that to say the field is worthless. I did read through many papers in the referenced journals and it seems really surprising to me. I’d love to see some of the research reproduced. But I’m not even sure how you could scientifically prove a field worthless.
- ATsch 8y ago
  > I don’t get this at all from the authors. My take is that they call out serious flaws in the rigor of how papers are reviewed. I don’t think it’s reasonable to extend that to say the field is worthless. I think it's rather undeniable that the Authors essay was not a good-faith attempt at criticizing the peer-review process, but an attempt to discredit certain fields that they didn't like, even by their own admission. I mean, the title of the essay alone should make this very clear. Here's some quotes nonetheless: > We spent that time writing academic papers and publishing them in respected peer-reviewed journals associated with fields of scholarship loosely known as “cultural studies” or “identity studies” > As a result of this work, we have come to call these fields “grievance studies” > We undertook this project to study, understand, and expose the reality of grievance studies, which is corrupting academic research > The biggest difference between us and the scholarship we are studying by emulation is that we know we made things up. > these fields of study do not continue the important and noble liberal work of the civil rights movements; they corrupt it while trading upon their good names to keep pushing a kind of social snake oil onto a public that keeps getting sicker [Note also that this essentially an academic way of phrasing the same things you'll hear people like the far-right Stefan Molyneux (on who's show one of the authors has been a guest several times) say.] Of importance here that it's not the scientific practices that are "corrupting academia" but the fact that these fields exist at all. I'm sure you can find more if you really want to. Compare this to say the "reproducibility in Psychology" study, which unsurprisingly doesn't dedicate half of their text to talking about why the authors think psychology is bad. Don't be mistaken: The whole thing was an attempt to cash in on the general crisis the scientific process is facing to score some easy political points and not a neutral and constructive attempt at improving the quality of research in the field. Or rather, their proposal at improving the quality would involve getting rid of the fields that they don't personally like.
pfortuny 8y ago
Oh, right: so you expect a “scientific experiiment”, with reproducibility. Yep. Right. OK. Come on.
jdietrich 8y ago
>The assertion of the authors is that this stuff getting published means the whole field is worthless. Which is of course ridiculous. Perhaps, but we're undoubtedly in the midst of a crisis of confidence in the process of scientific publication. From the reproducibility crisis to the Diederik Stapel affair to the paper mills in China, almost every field is being severely undermined by weak or deliberately fraudulent research. Frankly, I've almost given up on the scientific literature as a source of useful information, because I can't really trust most of it. The tools and processes of scientific publication just aren't fit for purpose. My Google Scholar and PubMed searches are overwhelmed with obviously junk papers from pay-to-play journals. Citation is no longer a credible metric because of citation rings. Established high-impact journals have reasonable quality standards, but they still get bamboozled with junk and a paper doesn't come with the supporting evidence I would need to evaluate its validity in any meaningful way. The signal-to-noise ratio of academic publishing is collapsing, and it's bringing down trust in the scientific method with it. We need major reform and we need it now, because the credibility of science as a whole is under dire threat. The AllTrials campaign has made significant gains in reducing publication bias in medical research, but we need to go much further. We need a publish-by-default model, where everything associated with a research project is published and failure to do so is treated as academic misconduct. We need gold open access, we need full datasets, we need source code, we need rough drafts and lab notes and reviewer's comments. Equally importantly, we need to fix the cultural pathology of publish-or-perish that's driving the proliferation of poor-quality research.

by ATsch 8y ago

prepend 8y ago

“The assertion of the authors is that this stuff getting published means the whole field is worthless.” I don’t get this at all from the authors. My take is that they call out serious flaws in the rigor of how papers are reviewed. I don’t think it’s reasonable to extend that to say the field is worthless. I did read through many papers in the referenced journals and it seems really surprising to me. I’d love to see some of the research reproduced. But I’m not even sure how you could scientifically prove a field worthless.

ATsch 8y ago
> I don’t get this at all from the authors. My take is that they call out serious flaws in the rigor of how papers are reviewed. I don’t think it’s reasonable to extend that to say the field is worthless. I think it's rather undeniable that the Authors essay was not a good-faith attempt at criticizing the peer-review process, but an attempt to discredit certain fields that they didn't like, even by their own admission. I mean, the title of the essay alone should make this very clear. Here's some quotes nonetheless: > We spent that time writing academic papers and publishing them in respected peer-reviewed journals associated with fields of scholarship loosely known as “cultural studies” or “identity studies” > As a result of this work, we have come to call these fields “grievance studies” > We undertook this project to study, understand, and expose the reality of grievance studies, which is corrupting academic research > The biggest difference between us and the scholarship we are studying by emulation is that we know we made things up. > these fields of study do not continue the important and noble liberal work of the civil rights movements; they corrupt it while trading upon their good names to keep pushing a kind of social snake oil onto a public that keeps getting sicker [Note also that this essentially an academic way of phrasing the same things you'll hear people like the far-right Stefan Molyneux (on who's show one of the authors has been a guest several times) say.] Of importance here that it's not the scientific practices that are "corrupting academia" but the fact that these fields exist at all. I'm sure you can find more if you really want to. Compare this to say the "reproducibility in Psychology" study, which unsurprisingly doesn't dedicate half of their text to talking about why the authors think psychology is bad. Don't be mistaken: The whole thing was an attempt to cash in on the general crisis the scientific process is facing to score some easy political points and not a neutral and constructive attempt at improving the quality of research in the field. Or rather, their proposal at improving the quality would involve getting rid of the fields that they don't personally like.

pfortuny 8y ago

Oh, right: so you expect a “scientific experiiment”, with reproducibility. Yep. Right. OK. Come on.

jdietrich 8y ago

>The assertion of the authors is that this stuff getting published means the whole field is worthless. Which is of course ridiculous. Perhaps, but we're undoubtedly in the midst of a crisis of confidence in the process of scientific publication. From the reproducibility crisis to the Diederik Stapel affair to the paper mills in China, almost every field is being severely undermined by weak or deliberately fraudulent research. Frankly, I've almost given up on the scientific literature as a source of useful information, because I can't really trust most of it. The tools and processes of scientific publication just aren't fit for purpose. My Google Scholar and PubMed searches are overwhelmed with obviously junk papers from pay-to-play journals. Citation is no longer a credible metric because of citation rings. Established high-impact journals have reasonable quality standards, but they still get bamboozled with junk and a paper doesn't come with the supporting evidence I would need to evaluate its validity in any meaningful way. The signal-to-noise ratio of academic publishing is collapsing, and it's bringing down trust in the scientific method with it. We need major reform and we need it now, because the credibility of science as a whole is under dire threat. The AllTrials campaign has made significant gains in reducing publication bias in medical research, but we need to go much further. We need a publish-by-default model, where everything associated with a research project is published and failure to do so is treated as academic misconduct. We need gold open access, we need full datasets, we need source code, we need rough drafts and lab notes and reviewer's comments. Equally importantly, we need to fix the cultural pathology of publish-or-perish that's driving the proliferation of poor-quality research.