4 ms·

It's not the editor's job to replicate the results. Imagine the same requirements in medical research, this would be crazy. Editors accept a paper based on the

It's not the editor's job to replicate the results. Imagine the same requirements in medical research, this would be crazy. Editors accept a paper based on the quality of the paper and the claim. Then the paper needs to be replicated a few times by independent research teams to be considered valid. This is the step we are probably missing in AI. Any published paper (and even unpublished, on arxiv) are considered valid and by the community but sometimes cannot be replicated.

Qworg 7y ago
This is a reasonable approach - a forcing function to require working code (social or otherwise) would be a good first step.
sampo 7y ago
> It's not the editor's job to replicate the results. They obviously meant the reviewers. Not everyone is intimately familiar with academic terminology.
- elcomet 7y ago
  The reviewers are commissioned by the editor, so I would say it's the same thing. Unless you meant the reviewers should replicate the papers with their own ressources, but I think it's not what we want. Reviewers are benevolent, and they don't have time nor resources to replicate results. Maybe a solution would be a platform, like a CI for machine learning where authors would send their codes, and the CI would run it for them, and make the results public. Then everyone could check the code and results, reviewers included.
  - p1esk 7y ago
    Please keep in mind that currently there are a lot of unqualified people reviewing papers, even at top conferences, as evidenced by OpenReview.net. There was an experiment done at NIPS a few years ago where something like half of the rejected papers got accepted after having been reviewed independently by different reviewers, and/or the other way around. If people can't even agree if paper is good or not, while assuming all results are reproducible, then we have a bigger problem. To me, getting the right people to review the paper is more important than reproducible code.
    - scott_s 7y ago
      There is a difference between unqualified people, and qualified but busy people. While there may be unqualified people reviewing papers, even qualified people who don't put in enough time and effort to review a paper comes to the same result. But the solution for that problem will look different. There's also a big difference between accepting a paper and judging if it is good. At a top conference like NIPS, lots of good papers will get rejected. That's the nature of being a top conference in a hot field. It's similar to a high school valedictorian not getting into Harvard: it's not a judgement that they are a bad student. But when you're an elite thing, and lots of people want in, you end up looking for reasons to reject rather than accept. You will end up rejecting good papers/students in such a case. NIPS, or NeurIPS now, is a particularly interesting example, as a colleague recently told me that they just got 12,000 submissions this year. There's also an issue in computer science papers where people may not like your paper based on personal taste. They may not be convinced you're solving a real problem, and if your paper involves a new artifact, the design of it. I refer to these as "your baby is ugly" reviews. They don't think you're wrong, they just don't like it. I think people's time is a bigger thing than the wrong people. One problem, I think, is that we tend to compete for these once-a-year conferences, causing an enormous time crunch for reviewers. VLDB has a great model where they do rolling acceptance throughout the year, accepting a few papers every month, even though the conference happens once a year (http://vldb.org/2019/?submission-guidelines http://vldb.org/2019/?submission-guidelines). I believe this is the best hybrid between the typical CS conference system, and the more common journal system in the rest of science and engineering.
      - p1esk 7y ago
        Submitting a paper to a conference for it to be published 11 months later is not going to work - the paper will be hopelessly obsolete by then. A better solution imo is to improve openreview.net process somehow, so that I'm motivated to go there, find papers relevant to my research (ideally get notified when such papers are posted for review), and leave a review (and perhaps vote on other reviews just like we do here on HN, or do something else to influence the acceptance decision). Obviously there should be methods to prevent abuse: moderators, reputations of reviewers, restricting reviewers to those with specific publications (e.g. based on keywords), etc. Btw, code reproducibility could be used to give some weight in such public review process.

Qworg 7y ago

This is a reasonable approach - a forcing function to require working code (social or otherwise) would be a good first step.

sampo 7y ago

> It's not the editor's job to replicate the results. They obviously meant the reviewers. Not everyone is intimately familiar with academic terminology.

elcomet 7y ago
The reviewers are commissioned by the editor, so I would say it's the same thing. Unless you meant the reviewers should replicate the papers with their own ressources, but I think it's not what we want. Reviewers are benevolent, and they don't have time nor resources to replicate results. Maybe a solution would be a platform, like a CI for machine learning where authors would send their codes, and the CI would run it for them, and make the results public. Then everyone could check the code and results, reviewers included.
- p1esk 7y ago
  Please keep in mind that currently there are a lot of unqualified people reviewing papers, even at top conferences, as evidenced by OpenReview.net. There was an experiment done at NIPS a few years ago where something like half of the rejected papers got accepted after having been reviewed independently by different reviewers, and/or the other way around. If people can't even agree if paper is good or not, while assuming all results are reproducible, then we have a bigger problem. To me, getting the right people to review the paper is more important than reproducible code.
  - scott_s 7y ago
    There is a difference between unqualified people, and qualified but busy people. While there may be unqualified people reviewing papers, even qualified people who don't put in enough time and effort to review a paper comes to the same result. But the solution for that problem will look different. There's also a big difference between accepting a paper and judging if it is good. At a top conference like NIPS, lots of good papers will get rejected. That's the nature of being a top conference in a hot field. It's similar to a high school valedictorian not getting into Harvard: it's not a judgement that they are a bad student. But when you're an elite thing, and lots of people want in, you end up looking for reasons to reject rather than accept. You will end up rejecting good papers/students in such a case. NIPS, or NeurIPS now, is a particularly interesting example, as a colleague recently told me that they just got 12,000 submissions this year. There's also an issue in computer science papers where people may not like your paper based on personal taste. They may not be convinced you're solving a real problem, and if your paper involves a new artifact, the design of it. I refer to these as "your baby is ugly" reviews. They don't think you're wrong, they just don't like it. I think people's time is a bigger thing than the wrong people. One problem, I think, is that we tend to compete for these once-a-year conferences, causing an enormous time crunch for reviewers. VLDB has a great model where they do rolling acceptance throughout the year, accepting a few papers every month, even though the conference happens once a year (http://vldb.org/2019/?submission-guidelines http://vldb.org/2019/?submission-guidelines). I believe this is the best hybrid between the typical CS conference system, and the more common journal system in the rest of science and engineering.
    - p1esk 7y ago
      Submitting a paper to a conference for it to be published 11 months later is not going to work - the paper will be hopelessly obsolete by then. A better solution imo is to improve openreview.net process somehow, so that I'm motivated to go there, find papers relevant to my research (ideally get notified when such papers are posted for review), and leave a review (and perhaps vote on other reviews just like we do here on HN, or do something else to influence the acceptance decision). Obviously there should be methods to prevent abuse: moderators, reputations of reviewers, restricting reviewers to those with specific publications (e.g. based on keywords), etc. Btw, code reproducibility could be used to give some weight in such public review process.