3 ms·

I'm active in the security and privacy field, and some conferences have started "artifact reviews" [1][2], where authors can optionally submit the data and code

I'm active in the security and privacy field, and some conferences have started "artifact reviews" [1][2], where authors can optionally submit the data and code that go along with their work. It's definitely an improvement, but the devil is in the details. Should these be mandatory? If they're mandatory, then what quality standards should artifacts meet? Who's ensuring those standards are followed? How should proprietary code and sensitive human-subjects research data be handled? There's also the question of what code should be made public: Is it just the analysis that produced the metrics on the paper (e.g., R scripts, Jupyter notebooks, etc.)? Or should it also include the data collection and pre-processing? How about the code that didn't produce immediately publishable results, but could be useful for future work? [1] https://petsymposium.org/artifacts.php https://petsymposium.org/artifacts.php [2] https://www.usenix.org/conference/usenixsecurity21/call-for-artifacts https://www.usenix.org/conference/usenixsecurity21/call-for-...

josalhor 5y ago
> where authors can optionally submit the data and code that go along with their work. So cool. I wish there were more steps in this direction. > Should these be mandatory? As an outsider, I would say no. However that should have a detrimental effect on the ability to get published if you are from an academic institution. I can totally see a gray area when it comes to non-academic publishers. > There's also the question of what code should be made public Shouldn't it be as much as possible? The way I see it is that the easier to replicate a paper, the more people can focus on its true utility. That is, by abstracting ourselves away over the details of the paper, what does this really contribute? From my point of view, publishing as much as possible maximizes the real-world impact of the research.

josalhor 5y ago

> where authors can optionally submit the data and code that go along with their work. So cool. I wish there were more steps in this direction. > Should these be mandatory? As an outsider, I would say no. However that should have a detrimental effect on the ability to get published if you are from an academic institution. I can totally see a gray area when it comes to non-academic publishers. > There's also the question of what code should be made public Shouldn't it be as much as possible? The way I see it is that the easier to replicate a paper, the more people can focus on its true utility. That is, by abstracting ourselves away over the details of the paper, what does this really contribute? From my point of view, publishing as much as possible maximizes the real-world impact of the research.