4 ms·

As an academic, nothing infuriates me more than the promise of code and/or data. There are valid reasons this happens, but the reality is such promise is quickl

As an academic, nothing infuriates me more than the promise of code and/or data. There are valid reasons this happens, but the reality is such promise is quickly forgotten. Even provided code is poorly documented and rarely works on a machine other than the author's. One common phrase is. "Data is available from the authors upon reasonable request" Try doing that for any paper older than 18 months. All it does it contribute to the replication crisis.

jiggawatts 3y ago
IMHO, publishing code for CS papers ought to be made a strict prerequisite for getting a paper in a journal. As in: you don’t get your name published unless you include a public Git repo URL and a docker container that can run your code. Otherwise, what’s to stop someone literally photoshopping some made up algorithm presented as vaguely believable pseudocode? “I have an invisible dragon in my garage, but I can’t open the door right now to show you. Just believe me.”
- was_a_dev 3y ago
  I was going to make this point, as I do agree. However in a fast paced field such as this, timing is crucial. And all of these papers so far are technically pre-prints on the arxiv.
  - reactordev 3y ago
    Which is kind of the point I was inferring. Everything now is just a stake in the ground, house pending… I think for a paper about an algorithm or novel way of doing something should be required to include the source code (not necessarily for release, but for proof to the journal reviewer at least). I’ve waited countless times for source code promised in a paper to never arrive, only to be flipped into some commercial offering only the ultra wealthy can afford. So I’m taking the stance of “I don’t believe you unless there’s a runtime or source code” because it’s too easy with AI to fake the claim.
    - was_a_dev 3y ago
      It does make me wonder how a reviewer can truly review a CS paper without running the code.

jiggawatts 3y ago

IMHO, publishing code for CS papers ought to be made a strict prerequisite for getting a paper in a journal. As in: you don’t get your name published unless you include a public Git repo URL and a docker container that can run your code. Otherwise, what’s to stop someone literally photoshopping some made up algorithm presented as vaguely believable pseudocode? “I have an invisible dragon in my garage, but I can’t open the door right now to show you. Just believe me.”

was_a_dev 3y ago
I was going to make this point, as I do agree. However in a fast paced field such as this, timing is crucial. And all of these papers so far are technically pre-prints on the arxiv.
- reactordev 3y ago
  Which is kind of the point I was inferring. Everything now is just a stake in the ground, house pending… I think for a paper about an algorithm or novel way of doing something should be required to include the source code (not necessarily for release, but for proof to the journal reviewer at least). I’ve waited countless times for source code promised in a paper to never arrive, only to be flipped into some commercial offering only the ultra wealthy can afford. So I’m taking the stance of “I don’t believe you unless there’s a runtime or source code” because it’s too easy with AI to fake the claim.
  - was_a_dev 3y ago
    It does make me wonder how a reviewer can truly review a CS paper without running the code.