3 ms·

Some ACM conferences are now handing out "badges" that mark papers where the associated "artifacts" are publicly available or have been independently reproduced

Some ACM conferences are now handing out "badges" that mark papers where the associated "artifacts" are publicly available or have been independently reproduced or replicated: https://www.acm.org/publications/policies/artifact-review-badging https://www.acm.org/publications/policies/artifact-review-ba... The main issue with that right now seems to be the lack of real incentives: those badges are nice to have, but in the end you're evaluated by publications in highly regarded journals and conferences, citations, and funding brought in. Those things of course tend to let people focus on novel, fancy, exciting work, instead of the time-consuming process of making things easy to replicate by others, or even replication of other people's experiments. Incentive-wise, the publication business follows the principle of fire-and-forget. What's needed, but I believe very difficult to introduce, is the strict expectation of independent replication. That would mean replication as part of the core review process, which would A. create a lot of work for reviewers B. initially reduce the attractiveness of whichever conference introduces it first. One thing that would be easy to introduce in computer science would be to make it mandatory to share all code used in experiments - if you cannot/won't do that, you should not able to take part in this game.

seanmcdirmid 7y ago
In my CS area, many results aren’t interesting enough to be replicated, even in high quality journals. Often the results aren’t even very relevant to what is otherwise a design paper, and are just there to check some boxes. The fact that research results aren’t replicated is just symptomatic of a much larger disfunction. It has been a long time since I’ve read a paper where I actually wanted to use whatever they were selling, let alone reproduce whatever results they were claiming. We don’t even have “novel, exciting, fancy” work! When something comes out of a paper that really changes the game (like say MapReduce), it gets replicated a lot.
- mafribe 7y ago
  what's your CS area?
- frankling_ 7y ago
  Good point, a lot of work really falls below the threshold where replication would still make sense. However, I do think there is a lot of incremental, piecemeal type of work that over the years could amount to a decent step forward. It's just that currently, from looking at the code behind quite a few published papers, the statements often just cannot be trusted enough to actually build on these works. It is my view that some subsubfields in CS sustain themselves by avoiding the most pertinent questions, because their answers would reveal that the entire subsubfield has been superseded, or was never that promising to begin with. Unfortunately, that type of noise generation is actually more profitable than a single paper saying "nope", although the value of the latter in terms of knowledge generation is enormously high.

seanmcdirmid 7y ago

In my CS area, many results aren’t interesting enough to be replicated, even in high quality journals. Often the results aren’t even very relevant to what is otherwise a design paper, and are just there to check some boxes. The fact that research results aren’t replicated is just symptomatic of a much larger disfunction. It has been a long time since I’ve read a paper where I actually wanted to use whatever they were selling, let alone reproduce whatever results they were claiming. We don’t even have “novel, exciting, fancy” work! When something comes out of a paper that really changes the game (like say MapReduce), it gets replicated a lot.

mafribe 7y ago
what's your CS area?
frankling_ 7y ago
Good point, a lot of work really falls below the threshold where replication would still make sense. However, I do think there is a lot of incremental, piecemeal type of work that over the years could amount to a decent step forward. It's just that currently, from looking at the code behind quite a few published papers, the statements often just cannot be trusted enough to actually build on these works. It is my view that some subsubfields in CS sustain themselves by avoiding the most pertinent questions, because their answers would reveal that the entire subsubfield has been superseded, or was never that promising to begin with. Unfortunately, that type of noise generation is actually more profitable than a single paper saying "nope", although the value of the latter in terms of knowledge generation is enormously high.