3 ms·

Papers Without Code

41 points by awb 6y ago

moab 6y ago
Seems about time that woke-tactics are applied to the high-stakes playground of academic reputation.
- moab 6y ago
  Snark aside, I can't see what good maintaining such a public list and encouraging people (presumably frustrated graduate students) to submit complaints. When working on the experimental section of a paper, maybe half of the papers you try to obtain some source code for or get help from the authors will end up being duds, and so you typically just say in your paper "we contacted the authors but unfortunately were not able to reproduce their experiments / obtain their source code". The existing approach seems like a more polite and equally effective way of encouraging authors to build reproducible research without resorting to McCarthy-style public punishment.
  - deleted 6y ago
    [deleted]
  - deleted 6y ago
    [deleted]
  - asdff 6y ago
    I agree that this naming and shaming doesn't do any good. Especially looking at this list, some of these are preprints. I think journals should start asking for all code used along with the data being deposited somewhere accessible to other researchers in order for the paper to be accepted. Ideally reviewers should have been asking for this, but they haven't, so it's up to the editors.
segfaultbuserr 6y ago
I see that currently http://papers.paperswithoutcode.com/ http://papers.paperswithoutcode.com/ lists 3 papers. I think the web interface needs more work. Now it has only a general-purpose spreadsheet and difficult to read. It also takes forever to load. For reference, they're: * Automating Bayesian optimization with Bayesian optimization > https://dl.acm.org/doi/10.5555/3327345.3327498 https://dl.acm.org/doi/10.5555/3327345.3327498 > Reason for submission: The original paper states: "Our code and data will be available online: https://github.com/gustavomalkomes/abo https://github.com/gustavomalkomes/abo." However, the repository does not contain the code. I have tried to contact the first author, however, he did not respond. > Reproduction: Weeks spent on reproduction > Authors notified: Yes - less than a week ago > Author response: Author has uploaded the code at: https://github.com/gustavomalkomes/abo https://github.com/gustavomalkomes/abo > Resolved ---- * A unified theory for the origin of grid cells through the lens of pattern formation Paper name > https://papers.nips.cc/paper/2019/hash/6e7d5d259be7bf56ed79029c4e621f44-Abstract.html https://papers.nips.cc/paper/2019/hash/6e7d5d259be7bf56ed790... > Reason for submission: We tried to reproduce the paper (very simple setup - RNN receiving 2D velocity inputs) and successfully implemented the model such that the loss converged to the same levels as reported in the paper. However, we failed to find the same representations that the authors focused on in their paper. We asked the authors for their code and they uploaded their code (https://github.com/ganguli-lab/grid-pattern-formation https://github.com/ganguli-lab/grid-pattern-formation) but running their code under their own hyperparameters does not yield their results. > Weeks spent on reproduction: 12 > Authors notified: Yes - more than 3 weeks ago ---- * Iterative Learning with Open-set Noisy Labels > https://arxiv.org/abs/1804.00092 https://arxiv.org/abs/1804.00092 > Reason for submission: I could not reproduce Yisen Wang's paper. I contacted a student at Michigan who compared to this work and said they could not reproduce it. The author was very dodgy in email exchanges. They used to have a github with code "coming soon" and deleted the github without code maybe a year later. I spent like a month trying to reproduce it. > Weeks spent on reproduction: 4 > Authors notified: Yes - more than 3 weeks ago
tehjoker 6y ago
The author of this site wishes to remain anonymous while purporting to act as an arbitor of others' reputations. Not a great look. They say they take reputations seriously, but also only promise to avoid taking spamming. Of course, it only takes one false accepted submission to potentially damage an author. This doesn't seem well thought out, well structured, or really intended to help the community -- more to punish people for not having code with their paper. People should have code, but this doesn't seem like it chief.
- belval 6y ago
  I understand your point but I wholeheartedly disagree. Being able to reproduce results in other fields can be a challenge due to high cost of hiring actual humans to experiment on. Most ML papers don't have that. You either have private datasets (understandable but you can still publish the code) or require huge amount of compute (cool project but you can still publish the code). If project like this allow me to know that the results of a paper cannot be reproduced then I am all for the name and shame approach. We are currently drowning in papers that claim to break SOTA and most are not reproducible. This is a waste of effort for everyone. This website will never amount to much anyway. It will be just a signal among the institution and previous work of the authors. With enough signals at some point we might be able to better ourselves as a scientific community.
flobosg 6y ago
See also the ReScience journal [1,2] and its Ten Years Reproducibility Challenge [3,4] from last year that tried to run scientific code written more than a decade ago. [1]: https://rescience.github.io/ https://rescience.github.io/ [2]: https://peerj.com/articles/cs-142/ https://peerj.com/articles/cs-142/ [3]: http://rescience.github.io/ten-years/ http://rescience.github.io/ten-years/ [4]: https://www.nature.com/articles/d41586-020-02462-7 https://www.nature.com/articles/d41586-020-02462-7
asdff 6y ago
I kinda take issue with the approach vs. simply attempting to email the corresponding author, but I understand the premise. I think the onus really has to start coming from journals. I should be able to have the pieces to reproduce a given work, but often times I'm left short of either the data (or a very key portion of it) or the code, and methods often just state what tools were used (much of it boilerplate if you look at the same authors other papers with the same tooling), not how these tools were used on the input data in particular nor how the output is handled. Reproducing a computational paper should be the easiest thing, but often times you are left in the weeds reading arcane documentation for tooling that hasn't been updated since that post doc left the group 6 years ago, and trying to piece together an appropriate pipeline, work that wouldn't have to be done if you could simply inspect the underlying code behind the figures presented in the text and see how these tools were used. I don't care if the pile of scripts is a sloppy mess, just give me a github repo with all the code used and a link to access the data from a public repository if it isn't under some form of controlled access (of which I should be able to easily apply for and also use for research purposes just like the original authors have done). Journals should mandate that all code and data used to generate results be made available to the community, because it's clear that with no mandate in place people aren't exactly taking this onus on themselves, nor do reviewers seem to be asking for this information from authors.

Papers Without Code

41 points by awb 6y ago

moab 6y ago

Seems about time that woke-tactics are applied to the high-stakes playground of academic reputation.

moab 6y ago
Snark aside, I can't see what good maintaining such a public list and encouraging people (presumably frustrated graduate students) to submit complaints. When working on the experimental section of a paper, maybe half of the papers you try to obtain some source code for or get help from the authors will end up being duds, and so you typically just say in your paper "we contacted the authors but unfortunately were not able to reproduce their experiments / obtain their source code". The existing approach seems like a more polite and equally effective way of encouraging authors to build reproducible research without resorting to McCarthy-style public punishment.
- deleted 6y ago
  [deleted]
- deleted 6y ago
  [deleted]
- asdff 6y ago
  I agree that this naming and shaming doesn't do any good. Especially looking at this list, some of these are preprints. I think journals should start asking for all code used along with the data being deposited somewhere accessible to other researchers in order for the paper to be accepted. Ideally reviewers should have been asking for this, but they haven't, so it's up to the editors.

segfaultbuserr 6y ago

I see that currently http://papers.paperswithoutcode.com/ http://papers.paperswithoutcode.com/ lists 3 papers. I think the web interface needs more work. Now it has only a general-purpose spreadsheet and difficult to read. It also takes forever to load. For reference, they're: * Automating Bayesian optimization with Bayesian optimization > https://dl.acm.org/doi/10.5555/3327345.3327498 https://dl.acm.org/doi/10.5555/3327345.3327498 > Reason for submission: The original paper states: "Our code and data will be available online: https://github.com/gustavomalkomes/abo https://github.com/gustavomalkomes/abo." However, the repository does not contain the code. I have tried to contact the first author, however, he did not respond. > Reproduction: Weeks spent on reproduction > Authors notified: Yes - less than a week ago > Author response: Author has uploaded the code at: https://github.com/gustavomalkomes/abo https://github.com/gustavomalkomes/abo > Resolved ---- * A unified theory for the origin of grid cells through the lens of pattern formation Paper name > https://papers.nips.cc/paper/2019/hash/6e7d5d259be7bf56ed79029c4e621f44-Abstract.html https://papers.nips.cc/paper/2019/hash/6e7d5d259be7bf56ed790... > Reason for submission: We tried to reproduce the paper (very simple setup - RNN receiving 2D velocity inputs) and successfully implemented the model such that the loss converged to the same levels as reported in the paper. However, we failed to find the same representations that the authors focused on in their paper. We asked the authors for their code and they uploaded their code (https://github.com/ganguli-lab/grid-pattern-formation https://github.com/ganguli-lab/grid-pattern-formation) but running their code under their own hyperparameters does not yield their results. > Weeks spent on reproduction: 12 > Authors notified: Yes - more than 3 weeks ago ---- * Iterative Learning with Open-set Noisy Labels > https://arxiv.org/abs/1804.00092 https://arxiv.org/abs/1804.00092 > Reason for submission: I could not reproduce Yisen Wang's paper. I contacted a student at Michigan who compared to this work and said they could not reproduce it. The author was very dodgy in email exchanges. They used to have a github with code "coming soon" and deleted the github without code maybe a year later. I spent like a month trying to reproduce it. > Weeks spent on reproduction: 4 > Authors notified: Yes - more than 3 weeks ago

tehjoker 6y ago

The author of this site wishes to remain anonymous while purporting to act as an arbitor of others' reputations. Not a great look. They say they take reputations seriously, but also only promise to avoid taking spamming. Of course, it only takes one false accepted submission to potentially damage an author. This doesn't seem well thought out, well structured, or really intended to help the community -- more to punish people for not having code with their paper. People should have code, but this doesn't seem like it chief.

belval 6y ago
I understand your point but I wholeheartedly disagree. Being able to reproduce results in other fields can be a challenge due to high cost of hiring actual humans to experiment on. Most ML papers don't have that. You either have private datasets (understandable but you can still publish the code) or require huge amount of compute (cool project but you can still publish the code). If project like this allow me to know that the results of a paper cannot be reproduced then I am all for the name and shame approach. We are currently drowning in papers that claim to break SOTA and most are not reproducible. This is a waste of effort for everyone. This website will never amount to much anyway. It will be just a signal among the institution and previous work of the authors. With enough signals at some point we might be able to better ourselves as a scientific community.

flobosg 6y ago

See also the ReScience journal [1,2] and its Ten Years Reproducibility Challenge [3,4] from last year that tried to run scientific code written more than a decade ago. [1]: https://rescience.github.io/ https://rescience.github.io/ [2]: https://peerj.com/articles/cs-142/ https://peerj.com/articles/cs-142/ [3]: http://rescience.github.io/ten-years/ http://rescience.github.io/ten-years/ [4]: https://www.nature.com/articles/d41586-020-02462-7 https://www.nature.com/articles/d41586-020-02462-7

asdff 6y ago

I kinda take issue with the approach vs. simply attempting to email the corresponding author, but I understand the premise. I think the onus really has to start coming from journals. I should be able to have the pieces to reproduce a given work, but often times I'm left short of either the data (or a very key portion of it) or the code, and methods often just state what tools were used (much of it boilerplate if you look at the same authors other papers with the same tooling), not how these tools were used on the input data in particular nor how the output is handled. Reproducing a computational paper should be the easiest thing, but often times you are left in the weeds reading arcane documentation for tooling that hasn't been updated since that post doc left the group 6 years ago, and trying to piece together an appropriate pipeline, work that wouldn't have to be done if you could simply inspect the underlying code behind the figures presented in the text and see how these tools were used. I don't care if the pile of scripts is a sloppy mess, just give me a github repo with all the code used and a link to access the data from a public repository if it isn't under some form of controlled access (of which I should be able to easily apply for and also use for research purposes just like the original authors have done). Journals should mandate that all code and data used to generate results be made available to the community, because it's clear that with no mandate in place people aren't exactly taking this onus on themselves, nor do reviewers seem to be asking for this information from authors.