4 ms·

Ten takeaways from ten years at Retraction Watch

60 points by severine 6y ago

exolymph 6y ago
> doing the right thing pays, which is why we created a category to recognize such actions and even helped start a now-moribund award, called the DIRT, to honor good behavior. If the incentives are wrong, do your bit to change the incentives! Retraction Watch is a worthy contribution to the repositories of shared knowledge that are of so much benefit to all of us.
elahieh 6y ago
I do have two anecdotes on this - came across two papers in my exact area of combinatorics that had incorrect results; but they had been submitted to IEEE conferences. So presumably the referees just glanced at them and waved them through. I wrote to the conference organizers explaining in very simple terms that it could be easily shown the results were wrong (i.e. "please send this to the original referees" - easy, you would think ...) One of them was sympathetic but said it can't be withdrawn from IEEE, and the other was like "get lost, I don't care". It was quite an eye opener. You really don't know what this is like until you've tried it. I eventually wrote to the author and his work (Porton Down) saying the same thing and the author reacted quite well. Still, the IEEE site for the papers doesn't say anything. Just my comments on pubpeer. And that doesn't come up in a google search. So there are still people wasting their time reading a result known to be wrong. https://pubpeer.com/publications/9842D9E91821B0F7CADE333BE9DC7B https://pubpeer.com/publications/9842D9E91821B0F7CADE333BE9D... https://pubpeer.com/publications/BF67542504DA9118ECBE8869EAF627 https://pubpeer.com/publications/BF67542504DA9118ECBE8869EAF... The only (obvious) red flag for a potential reader is of course - it was supposedly a very big result, so why was it at an IEEE conference instead of a top combinatorics journal?
- jashephe 6y ago
  The above is a good example of how results can be "wrong" in theory papers, but I work in an experimental field of research and at least in my world, having a paper present results that turn out to be wrong isn't really all that surprising. Experimental papers aren't proof of phenomena as much as they're an attempt to persuade you of the authors' worldview, which may or may not turn out to be correct in hindsight. This isn't grounds for retraction — there are plenty of well-intentioned and reasonably-interpreted experiments that end up not holding up as the field moves forward. Again, not saying that papers can't have problems with methodology, assumptions, or — particularly in the case of theory — soundness, but sometimes, "good" papers can be wrong. These papers sit in the body of literature, and are important context for modern findings. I do wonder if there could be some way to indicate to readers without domain-specific expertise that such a paper has been superseded, so to speak.
- _delirium 6y ago
  The norm for conference-driven fields, I think, is that formal retractions of proceedings papers are really only for outright misconduct, like plagiarism or fabricated data, not for mistakes, oversights, bad experimental design, etc. The proceedings are seen more as a record of an ongoing conversation that continues annually [1]. If you find something wrong, misleading, incomplete, etc. in a conference paper, the normal solution is that you write a reply conference paper criticizing/correcting the earlier paper. This does mean that if you dig up a paper from 10 or 20 years ago, you need to look through more recent literature to see if anyone's published criticisms. [1] This happens in practice in some conferences more than others of course. There are lower-tier conferences where people just dump papers and no kind of actual conversation or intellectual exchange is happening.
- specialist 6y ago
  Thank you for sharing. "...and the other was like "get lost, I don't care". It was quite an eye opener. You really don't know what this is like until you've tried it." Mirrors my experience in policy work and legislation. My current 3 takeaways: Attention economy rules all. I now see everyone, every org as information processors struggling with infoglut. So it's just very hard to even get noticed. Necessary triage makes course corrections very expensive. Long term, focus on improving feedback loops, reducing transaction costs. Applying those Deming and Drucker style organizational learning ideas to new domains. (I've had very limited success. See point #1.) Pace yourself, stay focused. Improvement is prohibitively expensive. Mark Twain, Brandolini, and many others, have marveled how silliness defeats truth. Frankly, I burned out. As David I sprinted whereas Goliath was running a marathon. And Goliath has mastered distraction, flooding the zone, provoking a response, patience. https://en.wikipedia.org/wiki/Brandolini%27s_law https://en.wikipedia.org/wiki/Brandolini%27s_law
bkandel 6y ago
> It’s not Big Pharma that’s responsible for most retractions This is not at all surprising to me. Big Pharma has teams of professional statisticians, writers, copy-editors, fact-checkers, etc. and have much more rigorously enforced standards than most academic research institutions. Their bias comes out more in how they frame the study and in the choice of things they study, but I think in general I would trust Big Pharma to be more accurate in reporting their study results than academic labs.
- yummypaint 6y ago
  Enjoy your thalidomide. Pharma has absolutely no intrinsic interest in scientific integrity or integrity in general. They exist solely to make money, and any cultivated appearance of good will is exactly that. They gobble up scientific talent as a means to that end, but R&D will always be a cost center for them. I would argue the massive profit-driven incentives to reach certain results are more insidious than a small research group nearing the end of a grant cycle. Given the historical context, it seems more likely that pharma is just better at concealing misconduct.
  - bkandel 6y ago
    I don't think we're disagreeing, and I agree that profits rather than human benefit drive pharma's research agenda. I'm just saying that if a paper from Big Pharma says a factual statement like "Reference X says Y" or "X% of patients had blood pressure of Y", I would expect Big Pharma research teams to have done a thorough job of checking to make sure they got it right.
dumbneurologist 6y ago
Is anyone aware of a "false positive" retraction? Meaning a retraction that was later found to be in error, resulting in re-publication (or whatever)? I imagine the bar for retraction is so high it would never happen, but it would be really interesting if it ever did.

60 points by severine 6y ago

> doing the right thing pays, which is why we created a category to recognize such actions and even helped start a now-moribund award, called the DIRT, to honor good behavior. If the incentives are wrong, do your bit to change the incentives! Retraction Watch is a worthy contribution to the repositories of shared knowledge that are of so much benefit to all of us.

elahieh 6y ago

I do have two anecdotes on this - came across two papers in my exact area of combinatorics that had incorrect results; but they had been submitted to IEEE conferences. So presumably the referees just glanced at them and waved them through. I wrote to the conference organizers explaining in very simple terms that it could be easily shown the results were wrong (i.e. "please send this to the original referees" - easy, you would think ...) One of them was sympathetic but said it can't be withdrawn from IEEE, and the other was like "get lost, I don't care". It was quite an eye opener. You really don't know what this is like until you've tried it. I eventually wrote to the author and his work (Porton Down) saying the same thing and the author reacted quite well. Still, the IEEE site for the papers doesn't say anything. Just my comments on pubpeer. And that doesn't come up in a google search. So there are still people wasting their time reading a result known to be wrong. https://pubpeer.com/publications/9842D9E91821B0F7CADE333BE9DC7B https://pubpeer.com/publications/9842D9E91821B0F7CADE333BE9D... https://pubpeer.com/publications/BF67542504DA9118ECBE8869EAF627 https://pubpeer.com/publications/BF67542504DA9118ECBE8869EAF... The only (obvious) red flag for a potential reader is of course - it was supposedly a very big result, so why was it at an IEEE conference instead of a top combinatorics journal?

jashephe 6y ago
The above is a good example of how results can be "wrong" in theory papers, but I work in an experimental field of research and at least in my world, having a paper present results that turn out to be wrong isn't really all that surprising. Experimental papers aren't proof of phenomena as much as they're an attempt to persuade you of the authors' worldview, which may or may not turn out to be correct in hindsight. This isn't grounds for retraction — there are plenty of well-intentioned and reasonably-interpreted experiments that end up not holding up as the field moves forward. Again, not saying that papers can't have problems with methodology, assumptions, or — particularly in the case of theory — soundness, but sometimes, "good" papers can be wrong. These papers sit in the body of literature, and are important context for modern findings. I do wonder if there could be some way to indicate to readers without domain-specific expertise that such a paper has been superseded, so to speak.
_delirium 6y ago
The norm for conference-driven fields, I think, is that formal retractions of proceedings papers are really only for outright misconduct, like plagiarism or fabricated data, not for mistakes, oversights, bad experimental design, etc. The proceedings are seen more as a record of an ongoing conversation that continues annually [1]. If you find something wrong, misleading, incomplete, etc. in a conference paper, the normal solution is that you write a reply conference paper criticizing/correcting the earlier paper. This does mean that if you dig up a paper from 10 or 20 years ago, you need to look through more recent literature to see if anyone's published criticisms. [1] This happens in practice in some conferences more than others of course. There are lower-tier conferences where people just dump papers and no kind of actual conversation or intellectual exchange is happening.
specialist 6y ago
Thank you for sharing. "...and the other was like "get lost, I don't care". It was quite an eye opener. You really don't know what this is like until you've tried it." Mirrors my experience in policy work and legislation. My current 3 takeaways: Attention economy rules all. I now see everyone, every org as information processors struggling with infoglut. So it's just very hard to even get noticed. Necessary triage makes course corrections very expensive. Long term, focus on improving feedback loops, reducing transaction costs. Applying those Deming and Drucker style organizational learning ideas to new domains. (I've had very limited success. See point #1.) Pace yourself, stay focused. Improvement is prohibitively expensive. Mark Twain, Brandolini, and many others, have marveled how silliness defeats truth. Frankly, I burned out. As David I sprinted whereas Goliath was running a marathon. And Goliath has mastered distraction, flooding the zone, provoking a response, patience. https://en.wikipedia.org/wiki/Brandolini%27s_law https://en.wikipedia.org/wiki/Brandolini%27s_law

bkandel 6y ago

> It’s not Big Pharma that’s responsible for most retractions This is not at all surprising to me. Big Pharma has teams of professional statisticians, writers, copy-editors, fact-checkers, etc. and have much more rigorously enforced standards than most academic research institutions. Their bias comes out more in how they frame the study and in the choice of things they study, but I think in general I would trust Big Pharma to be more accurate in reporting their study results than academic labs.

yummypaint 6y ago
Enjoy your thalidomide. Pharma has absolutely no intrinsic interest in scientific integrity or integrity in general. They exist solely to make money, and any cultivated appearance of good will is exactly that. They gobble up scientific talent as a means to that end, but R&D will always be a cost center for them. I would argue the massive profit-driven incentives to reach certain results are more insidious than a small research group nearing the end of a grant cycle. Given the historical context, it seems more likely that pharma is just better at concealing misconduct.
- bkandel 6y ago
  I don't think we're disagreeing, and I agree that profits rather than human benefit drive pharma's research agenda. I'm just saying that if a paper from Big Pharma says a factual statement like "Reference X says Y" or "X% of patients had blood pressure of Y", I would expect Big Pharma research teams to have done a thorough job of checking to make sure they got it right.

dumbneurologist 6y ago

Is anyone aware of a "false positive" retraction? Meaning a retraction that was later found to be in error, resulting in re-publication (or whatever)? I imagine the bar for retraction is so high it would never happen, but it would be really interesting if it ever did.