9 ms·

I do have two anecdotes on this - came across two papers in my exact area of combinatorics that had incorrect results; but they had been submitted to IEEE confe

I do have two anecdotes on this - came across two papers in my exact area of combinatorics that had incorrect results; but they had been submitted to IEEE conferences. So presumably the referees just glanced at them and waved them through. I wrote to the conference organizers explaining in very simple terms that it could be easily shown the results were wrong (i.e. "please send this to the original referees" - easy, you would think ...) One of them was sympathetic but said it can't be withdrawn from IEEE, and the other was like "get lost, I don't care". It was quite an eye opener. You really don't know what this is like until you've tried it. I eventually wrote to the author and his work (Porton Down) saying the same thing and the author reacted quite well. Still, the IEEE site for the papers doesn't say anything. Just my comments on pubpeer. And that doesn't come up in a google search. So there are still people wasting their time reading a result known to be wrong. https://pubpeer.com/publications/9842D9E91821B0F7CADE333BE9DC7B https://pubpeer.com/publications/9842D9E91821B0F7CADE333BE9D... https://pubpeer.com/publications/BF67542504DA9118ECBE8869EAF627 https://pubpeer.com/publications/BF67542504DA9118ECBE8869EAF... The only (obvious) red flag for a potential reader is of course - it was supposedly a very big result, so why was it at an IEEE conference instead of a top combinatorics journal?

jashephe 6y ago
The above is a good example of how results can be "wrong" in theory papers, but I work in an experimental field of research and at least in my world, having a paper present results that turn out to be wrong isn't really all that surprising. Experimental papers aren't proof of phenomena as much as they're an attempt to persuade you of the authors' worldview, which may or may not turn out to be correct in hindsight. This isn't grounds for retraction — there are plenty of well-intentioned and reasonably-interpreted experiments that end up not holding up as the field moves forward. Again, not saying that papers can't have problems with methodology, assumptions, or — particularly in the case of theory — soundness, but sometimes, "good" papers can be wrong. These papers sit in the body of literature, and are important context for modern findings. I do wonder if there could be some way to indicate to readers without domain-specific expertise that such a paper has been superseded, so to speak.
_delirium 6y ago
The norm for conference-driven fields, I think, is that formal retractions of proceedings papers are really only for outright misconduct, like plagiarism or fabricated data, not for mistakes, oversights, bad experimental design, etc. The proceedings are seen more as a record of an ongoing conversation that continues annually [1]. If you find something wrong, misleading, incomplete, etc. in a conference paper, the normal solution is that you write a reply conference paper criticizing/correcting the earlier paper. This does mean that if you dig up a paper from 10 or 20 years ago, you need to look through more recent literature to see if anyone's published criticisms. [1] This happens in practice in some conferences more than others of course. There are lower-tier conferences where people just dump papers and no kind of actual conversation or intellectual exchange is happening.
specialist 6y ago
Thank you for sharing. "...and the other was like "get lost, I don't care". It was quite an eye opener. You really don't know what this is like until you've tried it." Mirrors my experience in policy work and legislation. My current 3 takeaways: Attention economy rules all. I now see everyone, every org as information processors struggling with infoglut. So it's just very hard to even get noticed. Necessary triage makes course corrections very expensive. Long term, focus on improving feedback loops, reducing transaction costs. Applying those Deming and Drucker style organizational learning ideas to new domains. (I've had very limited success. See point #1.) Pace yourself, stay focused. Improvement is prohibitively expensive. Mark Twain, Brandolini, and many others, have marveled how silliness defeats truth. Frankly, I burned out. As David I sprinted whereas Goliath was running a marathon. And Goliath has mastered distraction, flooding the zone, provoking a response, patience. https://en.wikipedia.org/wiki/Brandolini%27s_law https://en.wikipedia.org/wiki/Brandolini%27s_law

jashephe 6y ago

The above is a good example of how results can be "wrong" in theory papers, but I work in an experimental field of research and at least in my world, having a paper present results that turn out to be wrong isn't really all that surprising. Experimental papers aren't proof of phenomena as much as they're an attempt to persuade you of the authors' worldview, which may or may not turn out to be correct in hindsight. This isn't grounds for retraction — there are plenty of well-intentioned and reasonably-interpreted experiments that end up not holding up as the field moves forward. Again, not saying that papers can't have problems with methodology, assumptions, or — particularly in the case of theory — soundness, but sometimes, "good" papers can be wrong. These papers sit in the body of literature, and are important context for modern findings. I do wonder if there could be some way to indicate to readers without domain-specific expertise that such a paper has been superseded, so to speak.

_delirium 6y ago

The norm for conference-driven fields, I think, is that formal retractions of proceedings papers are really only for outright misconduct, like plagiarism or fabricated data, not for mistakes, oversights, bad experimental design, etc. The proceedings are seen more as a record of an ongoing conversation that continues annually [1]. If you find something wrong, misleading, incomplete, etc. in a conference paper, the normal solution is that you write a reply conference paper criticizing/correcting the earlier paper. This does mean that if you dig up a paper from 10 or 20 years ago, you need to look through more recent literature to see if anyone's published criticisms. [1] This happens in practice in some conferences more than others of course. There are lower-tier conferences where people just dump papers and no kind of actual conversation or intellectual exchange is happening.

specialist 6y ago

Thank you for sharing. "...and the other was like "get lost, I don't care". It was quite an eye opener. You really don't know what this is like until you've tried it." Mirrors my experience in policy work and legislation. My current 3 takeaways: Attention economy rules all. I now see everyone, every org as information processors struggling with infoglut. So it's just very hard to even get noticed. Necessary triage makes course corrections very expensive. Long term, focus on improving feedback loops, reducing transaction costs. Applying those Deming and Drucker style organizational learning ideas to new domains. (I've had very limited success. See point #1.) Pace yourself, stay focused. Improvement is prohibitively expensive. Mark Twain, Brandolini, and many others, have marveled how silliness defeats truth. Frankly, I burned out. As David I sprinted whereas Goliath was running a marathon. And Goliath has mastered distraction, flooding the zone, provoking a response, patience. https://en.wikipedia.org/wiki/Brandolini%27s_law https://en.wikipedia.org/wiki/Brandolini%27s_law