7 ms·

More Wrong Things I Said in Papers

97 points by ikeboy 10y ago

startupdiscuss 10y ago
You know, these days, we shouldn't have the concept of a blog or paper being published. It should be collaborative effort. The initial post should be a comment period where you ask for feedback and no matter how large your mistake, it should be considered okay. And slowly as you get more eyes on it, it should solidify into a more accepted paper.
- jacquesm 10y ago
  A large part of science drives on ego, the process you describe would reduce that considerably. 'Publish or perish' works if there are a very small number of authors at the top of a paper.
  - lgas 10y ago
    Great, let's change that. Letting ego drive everything does not seem like a good idea.
    - DannyBee 10y ago
      Okay, let me know when you get us all to be robotic science-proving automatons.
    - tormeh 10y ago
      Dude, how will science progress without ego? That's like trying to remove greed from business. The remainder might have higher quality than what we have now, but the quantity would be tiny. It's not like scientists are in it for the money...
    - coldtea 10y ago
      Yeah, because such things as human nature we can just go ahead and change...
      - seanmcdirmid 10y ago
        If the communists could do it, we can to! Oh wait....
  - SilasX 10y ago
    I think you can have both -- keep a record of who made what suggestion when and which got merged into what branch on whose approval. Wiki/git-style tools already do (most of) this. In fact, that makes ego validation easier: "I contributed these insights, reviewed these contributions, and received these upvotes/contribution acceptances."
  - johncolanduoni 10y ago
    > small number of authors Have you seen papers in particle physics? Sometimes they don't fit on a single page!
  - johncolanduoni 10y ago
    The same way open source projects don't afford their collaborators any fame or recognition? Publish or perish still works as long as the history of contributions is clear (which seems like a requirement for a variety of reasons).
    - lwjones 10y ago
      I know several open source projects first hand where the founder gets all the recognition while most of the valuable parts were written by others. Of course you can see the contributions in the VCS history, but that doesn't translate to public recognition.
  - thaumasiotes 10y ago
    Different specialties have different authorship cultures; it is trivial to demonstrate that science goes on (in physics, no less!) when papers are headed by many, many authors. http://www.improbable.com/airchives/classical/articles/peanut_butter_rotation.html http://www.improbable.com/airchives/classical/articles/peanu...
- beambot 10y ago
  arXiv.org does a better job than most at allowing resubmissions... and serving as a repository for rebuttals that might not warrant journal publications. Lijie Chen (the student Scott mentioned) used arXiv.org's preprints for this purpose.
- zmb_ 10y ago
  I think an open system along these lines would result in too poor signal to noise ratio. There are far more people willing to engage in bikeshedding than people with the expertise, interest, time and funding to make valuable contributions. Today's system is already collaborative. It is precisely those people with the expertise, interest, time and funding in a particular problem that co-author the papers. Then there is the review process to provide feedback and spot the serious mistakes and problems with the papers. After which, for significant works, there are follow up papers by various authors to expand, refute or clarify the original work.
- amelius 10y ago
  I think we need, first and foremost, a good mechanism which allows people to comment on papers, ask questions, and (for authors) add errata. Right now, there are websites which allow people to discuss papers, but the problem is that a paper has no "home", so the discussion takes place on different sites, and therefore gets watered down. I guess we need to agree on a standard.
clifanatic 10y ago
I wish I was smart enough to be that wrong.
- deleted 10y ago
  [deleted]
- Noseshine 10y ago
  How do you know you are not? Well, you certainly have a case by making this assumption in the first place. Are you "not smart enough" in all areas, or just not not in (deep) math? Rhetorical question. Look at "IQ". There is no such thing as "IQ" - simply because there are so man vastly different things that you should measure. I forgot the lecture, but you can show that the exact same "IQ" can be for people with vastly different skill sets, or different IQs with a completely different distribution of abilities, and the higher IQ losing out in one or more fields. The guy with the Einstein math and physics abilities can be a total dimwit when it comes to managing or just dealing with people, for example, something I would argue the world is in much greater need of compared to getting more math geniuses. When you don't understand something ask yourself: Do I actually need to understand this? I took about 70 edX and Coursera courses mostly in completely different fields then my own (CS). One of the most important lessons is that there is sooooooo much knowledge. Have a look at this little story of a very simple product that humans make (ignore which concrete one they chose): https://medium.com/@kevin_ashton/what-coke-contains-221d449929ef https://medium.com/@kevin_ashton/what-coke-contains-221d4499... Summary quote (product name replaced with "X"): > The number of individuals who know how to make X is zero. > The number of individual nations that could produce X is zero. That's for one of the most simple products out there. You do not need to understand each and every subject. You are not dumb if you don't. You may be dumb if you think you are dumb that you don't understand every single random subject... :)
  - nabla9 10y ago
    >Look at "IQ". There is no such thing as "IQ" - simply because there are so man vastly different things that you should measure. When you measure vastly different things, like reaction time and performance in IQ tests, there is surprisingly strong correlation (> 0.8 without correction for attenuation). This supports the hypothesis that there is such thing as G-factor. None of those vastly different things we can measure measures it directly, but they all together point into the same direction. What has reaction time have to do with academic performance, job attainment, income, or IQ tests? Most of those who argue for different intelligences don't provide good evidense against strong correlation.
schoen 10y ago
I wrote a paper a few years ago that was published in a journal and which I felt was subsequently refuted by another paper. (I argued that for various reasons a server can never tell if a client has been virtualized or modified or not in the absence of hardware-based anti-virtualization features, but https://www.cs.cmu.edu/~jfrankli/hotos07/vmm_detection_hotos07.pdf https://www.cs.cmu.edu/~jfrankli/hotos07/vmm_detection_hotos... and some recently work on obfuscators make me think I was mostly to entirely wrong.) Particularly since I don't have a blog right now, I keep thinking that I have no way to correct my assertions in public. It feels like it would be useful to be able to do that, just so if anyone somehow comes across the old paper they could quickly find out why its arguments aren't valid.
- GFK_of_xmaspast 10y ago
  One usual approach is to write to the original journal and see if they'll post a notice.
- jmuhlich 10y ago
  PubPeer might be a decent option. They have some mechanism to identify comments from the author but I haven't investigated how it works. www.pubpeer.com
- anonymousDan 10y ago
  Personally I wouldn't worry about it, given it was an honest mistake, so long as more recent papers that refute your claim cite your paper. Whenever I'm reading an interesting paper I always do a reverse citation search to see what more recent papers say about it. I find that if a paper's claims are important enough someone in the field will publish a paper if they have results that contradict it.
- dredmorbius 10y ago
  Whatever happened to loyalty.org & virtuanova?
  - schoen 10y ago
    Some things happened that made me no longer feel that my life was "new", so I stopped blogging there. (One thing is that the friend's server that it was hosted on went down.) I had trouble thinking of a new blog name and so neglected to start a successor blog.
alecco 10y ago
http://retractionwatch.com/ http://retractionwatch.com/ is fun to visit once in a while
erdevs 10y ago
Aaronson is awesome. Intellectual honesty like this is to be admired in any field.
- 0xmohit 10y ago
  It's rare to see folks admit mistakes even in private. Writing it on a blog which is probably widely read is extremely rare. Such incidents are deeply inspiring.
dharmon 10y ago
In my PhD thesis I claimed a function was convex that was not. The embarrassing part was that in the next section I proved that an almost identical function was non-convex. Phew, I feel better getting that off my chest. Good thing nobody reads dissertations.

97 points by ikeboy 10y ago

You know, these days, we shouldn't have the concept of a blog or paper being published. It should be collaborative effort. The initial post should be a comment period where you ask for feedback and no matter how large your mistake, it should be considered okay. And slowly as you get more eyes on it, it should solidify into a more accepted paper.

jacquesm 10y ago
A large part of science drives on ego, the process you describe would reduce that considerably. 'Publish or perish' works if there are a very small number of authors at the top of a paper.
- lgas 10y ago
  Great, let's change that. Letting ego drive everything does not seem like a good idea.
  - DannyBee 10y ago
    Okay, let me know when you get us all to be robotic science-proving automatons.
  - tormeh 10y ago
    Dude, how will science progress without ego? That's like trying to remove greed from business. The remainder might have higher quality than what we have now, but the quantity would be tiny. It's not like scientists are in it for the money...
  - coldtea 10y ago
    Yeah, because such things as human nature we can just go ahead and change...
    - seanmcdirmid 10y ago
      If the communists could do it, we can to! Oh wait....
- SilasX 10y ago
  I think you can have both -- keep a record of who made what suggestion when and which got merged into what branch on whose approval. Wiki/git-style tools already do (most of) this. In fact, that makes ego validation easier: "I contributed these insights, reviewed these contributions, and received these upvotes/contribution acceptances."
- johncolanduoni 10y ago
  > small number of authors Have you seen papers in particle physics? Sometimes they don't fit on a single page!
- johncolanduoni 10y ago
  The same way open source projects don't afford their collaborators any fame or recognition? Publish or perish still works as long as the history of contributions is clear (which seems like a requirement for a variety of reasons).
  - lwjones 10y ago
    I know several open source projects first hand where the founder gets all the recognition while most of the valuable parts were written by others. Of course you can see the contributions in the VCS history, but that doesn't translate to public recognition.
- thaumasiotes 10y ago
  Different specialties have different authorship cultures; it is trivial to demonstrate that science goes on (in physics, no less!) when papers are headed by many, many authors. http://www.improbable.com/airchives/classical/articles/peanut_butter_rotation.html http://www.improbable.com/airchives/classical/articles/peanu...
beambot 10y ago
arXiv.org does a better job than most at allowing resubmissions... and serving as a repository for rebuttals that might not warrant journal publications. Lijie Chen (the student Scott mentioned) used arXiv.org's preprints for this purpose.
zmb_ 10y ago
I think an open system along these lines would result in too poor signal to noise ratio. There are far more people willing to engage in bikeshedding than people with the expertise, interest, time and funding to make valuable contributions. Today's system is already collaborative. It is precisely those people with the expertise, interest, time and funding in a particular problem that co-author the papers. Then there is the review process to provide feedback and spot the serious mistakes and problems with the papers. After which, for significant works, there are follow up papers by various authors to expand, refute or clarify the original work.
amelius 10y ago
I think we need, first and foremost, a good mechanism which allows people to comment on papers, ask questions, and (for authors) add errata. Right now, there are websites which allow people to discuss papers, but the problem is that a paper has no "home", so the discussion takes place on different sites, and therefore gets watered down. I guess we need to agree on a standard.

clifanatic 10y ago

I wish I was smart enough to be that wrong.

deleted 10y ago
[deleted]
Noseshine 10y ago
How do you know you are not? Well, you certainly have a case by making this assumption in the first place. Are you "not smart enough" in all areas, or just not not in (deep) math? Rhetorical question. Look at "IQ". There is no such thing as "IQ" - simply because there are so man vastly different things that you should measure. I forgot the lecture, but you can show that the exact same "IQ" can be for people with vastly different skill sets, or different IQs with a completely different distribution of abilities, and the higher IQ losing out in one or more fields. The guy with the Einstein math and physics abilities can be a total dimwit when it comes to managing or just dealing with people, for example, something I would argue the world is in much greater need of compared to getting more math geniuses. When you don't understand something ask yourself: Do I actually need to understand this? I took about 70 edX and Coursera courses mostly in completely different fields then my own (CS). One of the most important lessons is that there is sooooooo much knowledge. Have a look at this little story of a very simple product that humans make (ignore which concrete one they chose): https://medium.com/@kevin_ashton/what-coke-contains-221d449929ef https://medium.com/@kevin_ashton/what-coke-contains-221d4499... Summary quote (product name replaced with "X"): > The number of individuals who know how to make X is zero. > The number of individual nations that could produce X is zero. That's for one of the most simple products out there. You do not need to understand each and every subject. You are not dumb if you don't. You may be dumb if you think you are dumb that you don't understand every single random subject... :)
- nabla9 10y ago
  >Look at "IQ". There is no such thing as "IQ" - simply because there are so man vastly different things that you should measure. When you measure vastly different things, like reaction time and performance in IQ tests, there is surprisingly strong correlation (> 0.8 without correction for attenuation). This supports the hypothesis that there is such thing as G-factor. None of those vastly different things we can measure measures it directly, but they all together point into the same direction. What has reaction time have to do with academic performance, job attainment, income, or IQ tests? Most of those who argue for different intelligences don't provide good evidense against strong correlation.

schoen 10y ago

I wrote a paper a few years ago that was published in a journal and which I felt was subsequently refuted by another paper. (I argued that for various reasons a server can never tell if a client has been virtualized or modified or not in the absence of hardware-based anti-virtualization features, but https://www.cs.cmu.edu/~jfrankli/hotos07/vmm_detection_hotos07.pdf https://www.cs.cmu.edu/~jfrankli/hotos07/vmm_detection_hotos... and some recently work on obfuscators make me think I was mostly to entirely wrong.) Particularly since I don't have a blog right now, I keep thinking that I have no way to correct my assertions in public. It feels like it would be useful to be able to do that, just so if anyone somehow comes across the old paper they could quickly find out why its arguments aren't valid.

GFK_of_xmaspast 10y ago
One usual approach is to write to the original journal and see if they'll post a notice.
jmuhlich 10y ago
PubPeer might be a decent option. They have some mechanism to identify comments from the author but I haven't investigated how it works. www.pubpeer.com
anonymousDan 10y ago
Personally I wouldn't worry about it, given it was an honest mistake, so long as more recent papers that refute your claim cite your paper. Whenever I'm reading an interesting paper I always do a reverse citation search to see what more recent papers say about it. I find that if a paper's claims are important enough someone in the field will publish a paper if they have results that contradict it.
dredmorbius 10y ago
Whatever happened to loyalty.org & virtuanova?
- schoen 10y ago
  Some things happened that made me no longer feel that my life was "new", so I stopped blogging there. (One thing is that the friend's server that it was hosted on went down.) I had trouble thinking of a new blog name and so neglected to start a successor blog.

alecco 10y ago

http://retractionwatch.com/ http://retractionwatch.com/ is fun to visit once in a while

erdevs 10y ago

Aaronson is awesome. Intellectual honesty like this is to be admired in any field.

0xmohit 10y ago
It's rare to see folks admit mistakes even in private. Writing it on a blog which is probably widely read is extremely rare. Such incidents are deeply inspiring.

dharmon 10y ago

In my PhD thesis I claimed a function was convex that was not. The embarrassing part was that in the next section I proved that an almost identical function was non-convex. Phew, I feel better getting that off my chest. Good thing nobody reads dissertations.