3 ms·

From my experience as an undergraduate CS student this problem certainly does exist. I was once implementing an algorithm I've read about in a paper and could

by egis 15y ago

From my experience as an undergraduate CS student this problem certainly does exist. I was once implementing an algorithm I've read about in a paper and could not reproduce the published results. After lots of frustration I contacted the author just to find out that the associated dataset he published on his website was modified and is no longer suitable for testing of the original research. He also said he has lost the original dataset, which meant I could neither verify the correctness of my implementation, nor his published results...

amcintyre 15y ago
For some fields, it seems like the model should shift to "you will publish your source code and data alongside the writeup, else you won't be publishing here." Not doing so seems equivalent to publishing a paper about applications and characteristics of a new mathematical proof without actually including the proof.
- garenp 15y ago
  Source code should absolutely be required. If you can't reproduce results, you aren't doing science. I do not understand why publishers would even accept papers while _knowing_ that there is code out there that would verify the papers claims. The phrase _willfully ignorant_ comes to mind. Not too long ago I emailed someone at UIUC about a tool they wrote which was mentioned in a research paper I ran into online. I wanted to see if their method really was much better than previous ones, and if the trade-offs they made were worth it. Did I get it? No. Instead, they sent me a link to some new company founded based on the tool. I apparently had to be a "researcher" to receive the magical tarball. It also seemed to me to be a conflict of interest for a Professor to be working for a University and company at the same time - all while selectively choosing who can and cannot get access to their results.
alttag 15y ago
Some fields (including, as I understand, IEEE and ACM) have codes of ethics that provide minimum standards on years retention of study data. Such a response from a researcher should be extremely disconcerting.

by egis 15y ago

amcintyre 15y ago

For some fields, it seems like the model should shift to "you will publish your source code and data alongside the writeup, else you won't be publishing here." Not doing so seems equivalent to publishing a paper about applications and characteristics of a new mathematical proof without actually including the proof.

garenp 15y ago
Source code should absolutely be required. If you can't reproduce results, you aren't doing science. I do not understand why publishers would even accept papers while _knowing_ that there is code out there that would verify the papers claims. The phrase _willfully ignorant_ comes to mind. Not too long ago I emailed someone at UIUC about a tool they wrote which was mentioned in a research paper I ran into online. I wanted to see if their method really was much better than previous ones, and if the trade-offs they made were worth it. Did I get it? No. Instead, they sent me a link to some new company founded based on the tool. I apparently had to be a "researcher" to receive the magical tarball. It also seemed to me to be a conflict of interest for a Professor to be working for a University and company at the same time - all while selectively choosing who can and cannot get access to their results.

alttag 15y ago

Some fields (including, as I understand, IEEE and ACM) have codes of ethics that provide minimum standards on years retention of study data. Such a response from a researcher should be extremely disconcerting.