3 ms·

I'm not in a medical field, but the problem likely exists for our discipline as well. The issue, I suspect, stems from the nature of publishing: top-tier journ

by alttag 15y ago

I'm not in a medical field, but the problem likely exists for our discipline as well. The issue, I suspect, stems from the nature of publishing: top-tier journals only publish "interesting" research, which means reproducing research is less welcomed and if performed needs to be accompanied by a serious value-add. There is no incentive to reproduce. It makes it more difficult to publish. It doesn't lead to tenure. Why bother?

egis 15y ago
From my experience as an undergraduate CS student this problem certainly does exist. I was once implementing an algorithm I've read about in a paper and could not reproduce the published results. After lots of frustration I contacted the author just to find out that the associated dataset he published on his website was modified and is no longer suitable for testing of the original research. He also said he has lost the original dataset, which meant I could neither verify the correctness of my implementation, nor his published results...
- amcintyre 15y ago
  For some fields, it seems like the model should shift to "you will publish your source code and data alongside the writeup, else you won't be publishing here." Not doing so seems equivalent to publishing a paper about applications and characteristics of a new mathematical proof without actually including the proof.
  - garenp 15y ago
    Source code should absolutely be required. If you can't reproduce results, you aren't doing science. I do not understand why publishers would even accept papers while _knowing_ that there is code out there that would verify the papers claims. The phrase _willfully ignorant_ comes to mind. Not too long ago I emailed someone at UIUC about a tool they wrote which was mentioned in a research paper I ran into online. I wanted to see if their method really was much better than previous ones, and if the trade-offs they made were worth it. Did I get it? No. Instead, they sent me a link to some new company founded based on the tool. I apparently had to be a "researcher" to receive the magical tarball. It also seemed to me to be a conflict of interest for a Professor to be working for a University and company at the same time - all while selectively choosing who can and cannot get access to their results.
- alttag 15y ago
  Some fields (including, as I understand, IEEE and ACM) have codes of ethics that provide minimum standards on years retention of study data. Such a response from a researcher should be extremely disconcerting.
nooneelse 15y ago
I've been wondering lately if reproducing research could be one of the best "lines of attack" for productively moving more scientific discourse online to a open, reputation-tracking forum/repository. The big journals are likely to fight to keep a hold on their business model for new results. But a system of online Transactions of Repeated Results is not a domain they seem to really be serving now. So a small, possibly disruptive player could get some traction. It would seem a good fit for younger scientists at less prestigious (poorer) institutions, looking to get some form of objectively validated reputation they can cite for career reasons while they work to catch up to some part of the advancing-edge of research in their field. With enough traffic of scientists openly trying to reproduce the results in other journals (sometimes succeeding, sometimes not), it would be only a matter of time before some journals that have a bias toward "surprising" to the detriment of "true" get bitten by repeated results in the TRR that go against what they published. So long term, that would be a method by which TRR could garner some respect in fields.

by alttag 15y ago

egis 15y ago

From my experience as an undergraduate CS student this problem certainly does exist. I was once implementing an algorithm I've read about in a paper and could not reproduce the published results. After lots of frustration I contacted the author just to find out that the associated dataset he published on his website was modified and is no longer suitable for testing of the original research. He also said he has lost the original dataset, which meant I could neither verify the correctness of my implementation, nor his published results...

amcintyre 15y ago
For some fields, it seems like the model should shift to "you will publish your source code and data alongside the writeup, else you won't be publishing here." Not doing so seems equivalent to publishing a paper about applications and characteristics of a new mathematical proof without actually including the proof.
- garenp 15y ago
  Source code should absolutely be required. If you can't reproduce results, you aren't doing science. I do not understand why publishers would even accept papers while _knowing_ that there is code out there that would verify the papers claims. The phrase _willfully ignorant_ comes to mind. Not too long ago I emailed someone at UIUC about a tool they wrote which was mentioned in a research paper I ran into online. I wanted to see if their method really was much better than previous ones, and if the trade-offs they made were worth it. Did I get it? No. Instead, they sent me a link to some new company founded based on the tool. I apparently had to be a "researcher" to receive the magical tarball. It also seemed to me to be a conflict of interest for a Professor to be working for a University and company at the same time - all while selectively choosing who can and cannot get access to their results.
alttag 15y ago
Some fields (including, as I understand, IEEE and ACM) have codes of ethics that provide minimum standards on years retention of study data. Such a response from a researcher should be extremely disconcerting.

nooneelse 15y ago

I've been wondering lately if reproducing research could be one of the best "lines of attack" for productively moving more scientific discourse online to a open, reputation-tracking forum/repository. The big journals are likely to fight to keep a hold on their business model for new results. But a system of online Transactions of Repeated Results is not a domain they seem to really be serving now. So a small, possibly disruptive player could get some traction. It would seem a good fit for younger scientists at less prestigious (poorer) institutions, looking to get some form of objectively validated reputation they can cite for career reasons while they work to catch up to some part of the advancing-edge of research in their field. With enough traffic of scientists openly trying to reproduce the results in other journals (sometimes succeeding, sometimes not), it would be only a matter of time before some journals that have a bias toward "surprising" to the detriment of "true" get bitten by repeated results in the TRR that go against what they published. So long term, that would be a method by which TRR could garner some respect in fields.