11 ms·
The Arsenic DNA paper exposes flaws in peer review
- badman_ting 13y agoThe Last Psychiatrist talks in his/her blog sometimes about what a joke peer review is. Here is one entry: http://thelastpsychiatrist.com/2011/01/wakefield_and_the_autism_fraud.html http://thelastpsychiatrist.com/2011/01/wakefield_and_the_aut...
- judk 13y agoLP is so sarcastic and snarly that it is difficult to understand what they are trying to say. Not an effective tone in an article about the problems of misinformation.
- dekhn 13y agoHis title is a red herring. he admits later he didn't actually do this. His point is actually a rebuttal to John Bohannon's submission of a fake article to public/open journals, demonstrating the Science Magazine also makes the same poor errors in review as the open journals.
- IgorPartola 13y agoHere is a real case of this happening: http://en.wikipedia.org/wiki/Sokal_affair http://en.wikipedia.org/wiki/Sokal_affair
- jnbiche 13y agoDid you read the blog post? It discusses a much more update-to-date case of this happening, by a reported who just recently submitted a fake paper to a bunch of open-access journals, some of whom accepted it. He then wrote the experience up and submitted it to Science, who is spinning it as something typical of open-access journals. More more notable than the fact that some fly-by-night, pay-to-play "open access" journals accepted a scientifically and ethically dubious paper is the fact that there now exists a large body of established, high-quality open access journals who roundly rejected the paper and pointed out its many flaws. Meta publication bias strikes again.
- housel 13y agoThe point of the Sokal affair seems to lean more towards proving that postmodern cultural studies as a field is lacking in substance than to demonstrating the ineffectiveness of peer review in general.
- primelens 13y agoWhat bugs me about the Sokal affair is the implication that they didn't catch a bad review process but proved an entire discipline to be a sham (a discipline which they understood relatively little of to begin with). So, when you get junk accepted in a humanities journal, its because the field is about bullshitting anyway - when you get junk into a science journal it's a hitch with the review process?
- lmm 13y agoWas the Sokal paper published with inadequate review? Or was it passed by experts in the field?
- magicalist 13y agoAll it takes is reading the wikipedia article... But yes it was and no it didn't. That's always bugged me about the Sokal affair and people holding it up as an example of anything. While the editors did publish the paper, which is a basic problem in and of itself, they claim they wanted big revisions that Sokal refused to make, but then they published it anyway. Since the whole point of that journal at the time was to get wide-ranging viewpoints, I think there is some merit in the editor's claim of a simple betrayal of trust: they assumed Sokal was being straightforward, and the worst that would come of publishing his piece would be a bunch of letters telling him to stick to his day job. In any case, I don't know if we'll ever a definitive story of what really happened, how carefully they vetted the paper, etc. Regardless, the journal he submitted the paper to was not peer reviewed, so the whole thing demonstrates nothing about the peer review process.
- eli 13y agoIf you want to talk about honesty and integrity in publishing, you shouldn't give your blog post a blatantly misleading linkbait title.
- donohoe 13y agoTrue - but to make a very valid point about the state of peer review in science journals it is perfect.
- rubidium 13y ago``we need to get past the antiquated idea that the singular act of publication – or publication in a particular journal – should signal for all eternity that a paper is valid, let alone important'' Most scientists I know (physics) don't have this idea. They know publication is just a step. Most grad students learn at some point that just because it's published doesn't mean it's correct. Publication is a checkpoint, not a finish line, for any theory or experimental result.
- colechristensen 13y agoYou're absolutely correct, scientists don't suffer from any misconception that publishing equals truth. The general public, on the other hand, has little understanding of this.
- talmand 13y agoNor do the reporters and/or editorials that cite such peer-reviewed papers that agree with their opinions. Wait, maybe they do understand but they also understand that the general public doesn't understand.
- GrumpySimon 13y agoI think this is an artefact of how science is taught: In 1953 Watson and Crick published a paper on the structure of DNA. Everyone smiled and nodded. In my experience it's more like: Most scientists ignored it. Others were interested but had too much teaching to pay attention to it. A few people got pissed off and tried to prove it wrong. A few other people got excited and tried to build on it.
- GrumpySimon 13y agoAbsolutely. I see peer review as a way of - hopefully - filtering out the most egregious crap, and fixing the worst errors in papers while rounding them off. I feel all these debates are kind of pointless, because no-one I know treats a published article as gospel. We make up our own minds about the paper.
- 13y ago
- unreal37 13y agoHey Michael Eisen, we get it. You don't like the peer review journal process and having to pay for access to scientific research. If you don't like it, make a better system. Also -1 for having the worst form of linkbaity title, which is one that is utterly false and misleading, and lying for the first two paragraphs of that blog post.
- tedsanders 13y agoTo me, the title was obviously satire from the moment I read it. Also, he did make a better system. He co-founded the Public Library of Science (PLOS).
- judk 13y agoDoes PLOS protect against false claims somehow?
- rrmm 13y agoA better-ish system has already been made. The post mostly excoriates pay journals for implying that open access journals enable fraud and poor papers and pay journals prevent those sorts of papers. The arsenic-dna paper is used as a counterexample. The author instead says that ideas of peer review need to change. Although, everyone I know always thought peer review was a continuous process, more support for after-publication review would be nice.
- deleted 13y ago[deleted]
- zokier 13y ago> Any scientist can quickly point to dozens of papers – including, and perhaps especially, in high impact journals – that are deeply, deeply flawed – the arsenic DNA story is one of many recent examples Dozens of flawed papers out of how many total papers? Numbers are meaningless without a context.
- lutusp 13y agoQuote: "So I created a manuscript that claimed something extraordinary - that I’d discovered a bacteria that uses arsenic in its DNA instead of phosphorous [sic]." This tongue-in-cheek piece would be more plausible if the author could spell "phosphorus".
- piyush_soni 13y agoReally, get away from your computer screen for a while. Get a passport, visit places. Know about the rest of the world, what they wear, what they eat, how they talk, how they write.
- judk 13y agoWhere do they write "phosphorous"? They don't indiscriminately sprinkle 'u' after every 'o'. That would be "phosphourous".
- DanBC 13y agophousphourous? (https://en.wikipedia.org/wiki/Red_Phosphorous#Spelling_and_etymology https://en.wikipedia.org/wiki/Red_Phosphorous#Spelling_and_e...) > According to the Oxford English Dictionary, the correct spelling of the element is phosphorus. The word phosphorous is the adjectival form of the P3+ valence: so, just as sulfur forms sulfurous and sulfuric compounds, phosphorus forms phosphorous compounds (e.g., phosphorous acid) and P5+ valence phosphoric compounds (e.g., phosphoric acids and phosphates).
- lutusp 13y ago> Get a passport, visit places. I have: http://www.amazon.com/Confessions-Long-Distance-Sailor-Paul-Lutus/dp/1435710274/ http://www.amazon.com/Confessions-Long-Distance-Sailor-Paul-... If you had any idea to whom you're speaking, you would realize how utterly absurd your suggestion is. Speaking purely on a probabilistic basis, I know more about the world than you have any chance of knowing.
- skosuri 13y ago
- justncase80 13y agoThere should be an "Open Source" science journal. Where anyone can submit a paper and anyone can peer review or freely read said articles. User accounts could have a vetting process to verify their real world credentials and the articles they review would be weighted accordingly. Funding by donations and for-pay apis to do advanced searches perhaps? How awesome would it be to be able to freely read a repository of this kind of information? And to also be able to accurately say what is or isn't the current "scientific concensus". I feel like this shouldn't be too difficult to make.
- robotresearcher 13y agohttp://www.plosone.org/ http://www.plosone.org/ PLOS ONE is this. It is well respected already. The initial invited reviews are mainly a junk filter to establish baseline methodological quality and NOT to evaluate significance. From their "About us" page: "Unlike many journals which attempt to use the peer review process to determine whether or not an article reaches the level of 'importance' required by a given journal, PLOS ONE uses peer review to determine whether a paper is technically sound and worthy of inclusion in the published scientific record. Once the work is published in PLOS ONE, the broader community is then able to discuss and evaluate the significance of the article (through the number of citations it attracts; the downloads it achieves; the media and blog coverage it receives; and the post-publication Notes, Comments and Ratings that it receives on PLOS ONE etc)."
- jamesjporter 13y agoI'll just mention for those unaware that the author of the post is one of the founders of PLOS, which is an umbrella organization under which a number of journals (including PLOS ONE) are published.
- mbq 13y agoThis is something promising: http://www.peerageofscience.org/ http://www.peerageofscience.org/
- stopcodon 13y agoIn addition to the PLOS series of journals, there are also other modern open-access journals popping up like PeerJ (peerj.com) and elife (elife.elifesciences.org). Anyone can go on and download the papers, and usually the datasets used in the papers if you want to replicate their results.
- joshlegs 13y agopeer-review scientific analysis has been an issue for some time if you ask me. As the guy points out, all you have to do is find a few people already receptive to your idea. I'm not really sure why "peer-review" is so lofty a concept anyway.
- suprgeek 13y agoThis post takes the lead-in with the Felisa Wolfe-Simon "Arsenic DNA" paper. In recent times I remember this was one of the most Hyped-up papers something akin to the Cold fusion paper by Fleischmann and Pons. The contrast in the public response to the two incidents is fascinating. While Cold-fusion became a career-ender for anybody associated with it and Fleischmann & Pons both suffered pretty much banishment, repeated experiments have shown that there is something funny going on with that particular mix of compounds (Heavy water, Palladium, Nickel etc) - excess heat. With arsenic life however, follow-up experiments have proven that it was contamination that was a cause of the result. So there is NO ARSENIC based life. Yet the lead authors of pretty much a gigantic false claim -orchestrated shamelessly by NASA- seem to be doing pretty well, dodging the basic questions with that most famous catch-all - more work is needed (hence more money should be sent my way) to prove the (bogus) claims made.
- tgb 13y agoWarning! This title is a total lie! I know it's a lie done intentionally and calculatingly, but most of the time most people don't read past the title and will assume that it's true. I really think this needs to be edited! Even if it 'spoils' the article. Please do this, whoever can! At the time of writing the posted title here is the same as the article title: "I confess, I wrote the Arsenic DNA paper to expose flaws in peer-review at subscription based journals"
- mwc 13y agoIn legal practice, decisions are tested on appeal or by subsequent tribunals. While this process is currently conducted by paid-for editorial teams, most practitioners rely on citation tools which for any judgment provide links through to subsequent decisions which have, for example, "approved", "considered", "distinguished" (as in disapplied due to different facts, but this can have the practical effect of confining an earlier decision to its particular facts) or "overturned" that decision. Although Google Scholar seems to provide some authority measure through the "cited by" tool, I've often wondered why academia (or perhaps I've just missed it) hasn't developed a comparable "precedent" system for research.
- deleted 13y ago[deleted]
- jvdh 13y agoThere are 250+ journals that have cancer growth as part of their field? That shows you the problem right there. There is absolutely no way that a researcher in that field can keep up with publications in all of those journals.
- deleted 13y ago[deleted]
- utopkara 13y agoThe problem is not the opaque peer review system. Instead of making the reviews transparent, some journals solicit notes from other researchers to discuss the published articles. It is an ethical responsibility to publish notes which criticize previously published articles, regardless of how painful it is for the publisher to publish them. e.g. http://www.sciencemag.org/content/335/6069/665.summary http://www.sciencemag.org/content/335/6069/665.summary Reading an article that falsifies claims of another article is pure scientific joy. And such articles are not uncommon at all! The problem is, as pointed out in the beginning of the paragraph "we need to get past the antiquated idea that the singular act of publication – or publication in a particular journal – should signal for all eternity that a paper is valid, let alone important".