4 ms·

The day when code used to produce a paper must also be published can not come soon enough.

• goalieca 6y ago
  Arguably, data is just as important. Academics hoard their data and try to milk out every paper they can from it. The reward system is based on publishing as many papers as possible rather than just making a meaningful contribution.
  • jgeada 6y ago
    The reward system also prevents dead ends from being identified, publication of approaches that did not lead to the expected results or got nul results, publishing confirmations of prior papers, etc. Basically, the reward system is designed to be easy to measure and administer, but is not actually useful in any way to the advancement of science.
  • belval 6y ago
    Data is much trickier because your data source for medical, education or even just regular businesses don't want the added legal weight of making data freely available. This is obviously a shame, I was working on segmentation of open wounds and most papers include a "we are currently in talks with the hospital to make the data available". If you contact the authors directly they will tell you that their committee blocked it because the information is too sensitive.
    • abathur 6y ago
      It seems like there can be a balance between "the results are unverifiable because no one else can touch the data" and "effectively open-source the dataset"? Something like: "To make it easier to verify the code behind this paper, we've used <accepted standard project/practice> to generate a synthetic dataset with the same fields as the original and included it with the source code. The <data-owning institution> isn't comfortable with publishing the full dataset, but they did agree to provide the same data to groups working on verification studies as long as they're willing to sign a data privacy agreement. Send a query to <blahblahblah> ..."
      • belval 6y ago
        > but they did agree to provide the same data to groups working on verification studies as long as they're willing to sign a data privacy agreement. Send a query to <blahblahblah> ..." This would be administrative overhead, it will be shut down 9 times out of 10. I understand why this might seem easy but it really is not, you can have multiple hospital that each have their committee that agreed to give the researcher their data. They don't have a central authority that you can appeal to, much less someone that can green light your specific access. As for the synthetic datasets that's basically just having tests and was advocated for elsewhere in this thread.
        • abathur 6y ago
          > This would be administrative overhead I didn't say it was easy--I just said it struck a balance relative to trying to openly publish the whole dataset. Yes. Obviously comes with administrative overhead. So did dealing with the initial researcher. If the institution can manage the one, it can manage the other. > As for the synthetic datasets that's basically just having tests An appropriate synthetic dataset would inevitably be part of a great test suite, but it's also pretty simple to write narrow unit-tests that embed rather than stretch the same assumptions and biases that are also in the code (i.e., simple enough that even people who feed themselves with code do it). An independent project/practice for synthesizing sample datasets from the real dataset lowers the bar and clarifies the best-practice for releasing a dataset that a verifier could actually use to spot simple bugs, edge-cases, and algorithm issues. Ideally, yes, this practice nudges the researchers to bother running their program over generated sample datasets as well, and to pay attention to whether the results make sense.
• WanderPanda 6y ago
  Making this mandatory might have bad downstream effects like prohibiting publication of some research at all (GPT-X I am looking at you)
  • qppo 6y ago
    Closed source research isn't publication, it's advertisement.
    • WanderPanda 6y ago
      So R&D is not a thing, but A&D is? That would be new to me
• dandelion_lover 6y ago
  It won't happen until researchers are forced to do it. Please sign petition at https://publiccode.eu https://publiccode.eu and have a look at my other comment here.
• jhrmnn 6y ago
  In all my papers the results were produced on multiple days (spanning months), with multiple versions of the code, and they are computationally too expensive to reproduce with the final version of the code. I'm trying to keep track of all the used versions, but given that there is no automated framework for this (is there?) and research involves lots of experiments, it's never perfect. Given this context, any ideas how to do it better?
  • xen0 6y ago
    My first thought: Demand the journals provide hosting for a code repo that is part of your paper. For every numerical result, specify the version (e.g. a git tag) used to generate your result. And if that means scientists need to learn about version control, well... they should if they're writing code.
    • chriswarbo 6y ago
      I agree, except that AFAIK "tags" in git are not fixed, they can be deleted and re-created to point at a different commit. Hence I prefer to use (short) commit IDs, since changing them is infeasible.
      • xen0 6y ago
        I'm assuming the repo, once hosted and the paper is published, is "fixed" and cannot be changed by the authors. But commit ids work just as well.
    • mnw21cam 6y ago
      For a paper I recently submitted, the journal demanded a github release of the software.
    • jhrmnn 6y ago
      I should have mentioned that I of course use Git. But the need to manually keep track of the calculation~commit pairing is tiresome and error-prone.
  • 6y ago
• lambdatronics 6y ago
  I agree, with the caveat: publically-funded research.