4 ms·

The editors, previous to accepting. If it doesn't run out of the box and reproduce the result, it doesn't get published.

• p1esk 7y ago
  Some code might require 8 GPUs and a week to verify a single result. There could be a dozen results to be verified in a paper. Are you sure you have thought this through?
  • tsbinz 7y ago
    8 gpus and a week is cheap even, there are popular papers that need upwards of 10k GPU hours.
  • pfortuny 7y ago
    It is not like, ehem, Elsevier does not have the money to pay for this...
    • solidasparagus 7y ago
      AI papers do not get published by Elsevier
  • Qworg 7y ago
    I have thought it through. There are multiple major papers that are published each year that purport to show an advancement, but fail to demonstrate that advancement under different random parameters. This was a major topic at NIPS 2017, yet doesn't seem to be important enough to address.
  • sampo 7y ago
    Well, it usually takes months to get a paper reviewed anyway, so it's not like there isn't time for that...
  • anewhnaccount2 7y ago
    The experiments themselves should be conducted on fully logged public infrastructure so that it impossible to miss anything out. This final log would include the inputs: shell commands/git repositories/Dockerfiles, training and testing data; important intermediate data: Docker images and trained models; and outputs: tables of numbers or what have you. This way, the whole thing is vouched for automatically, and the editors don't need to do anything. It would be a big change, but this is actually possible, and it would make things easier for everyone down the line.
    • solidasparagus 7y ago
      Who is going to pay for that?
• elcomet 7y ago
  It's not the editor's job to replicate the results. Imagine the same requirements in medical research, this would be crazy. Editors accept a paper based on the quality of the paper and the claim. Then the paper needs to be replicated a few times by independent research teams to be considered valid. This is the step we are probably missing in AI. Any published paper (and even unpublished, on arxiv) are considered valid and by the community but sometimes cannot be replicated.
  • Qworg 7y ago
    This is a reasonable approach - a forcing function to require working code (social or otherwise) would be a good first step.
  • sampo 7y ago
    > It's not the editor's job to replicate the results. They obviously meant the reviewers. Not everyone is intimately familiar with academic terminology.
    • elcomet 7y ago
      The reviewers are commissioned by the editor, so I would say it's the same thing. Unless you meant the reviewers should replicate the papers with their own ressources, but I think it's not what we want. Reviewers are benevolent, and they don't have time nor resources to replicate results. Maybe a solution would be a platform, like a CI for machine learning where authors would send their codes, and the CI would run it for them, and make the results public. Then everyone could check the code and results, reviewers included.
      • p1esk 7y ago
        Please keep in mind that currently there are a lot of unqualified people reviewing papers, even at top conferences, as evidenced by OpenReview.net. There was an experiment done at NIPS a few years ago where something like half of the rejected papers got accepted after having been reviewed independently by different reviewers, and/or the other way around. If people can't even agree if paper is good or not, while assuming all results are reproducible, then we have a bigger problem. To me, getting the right people to review the paper is more important than reproducible code.
• kaitai 7y ago
  Agree with sibling commenters that editors shouldn't be doing this themselves -- in my field, for instance, an editor is just the paid-in-prestige prof who agrees to look through articles submitted in a certain area, find referees for the paper, cajole the referees to review it at 6 months, cajole the referees again at one year, etc., and then make a decision based on the referee reports. (Clearly I'm not in CS as a 1.5-year turnaround is not so nuts in my field.) Anyhow, I think Qworg has a point in that the editors do have responsibility for what they accept. It's true that referees do not have the resources to reproduce results: I for instance have limited time on the shared high-performance computing resources we have, so I'm not going to use it on someone else's paper. However, as another commenter says, this could be paid for. A journal could potentially "sell" this as a competitive advantage: papers in this journal were actually reproduced, by paid staff who are neither editors nor referees, people paid in currency that can be exchanged for goods (not academic currency).