4 ms·

>Failure to distribute the data and code not only greatly devalues the contributions of many scientists, it makes replication far more difficult, and opens the

>Failure to distribute the data and code not only greatly devalues the contributions of many scientists, it makes replication far more difficult, and opens the door for outright fabrication. Reasons why labs don't produce code: (1) Fear of being scooped; suppose you put in tons of man-hours into a custom population genetics association study for malaria in Africa, you don't want your competitor lab to sequence a bunch samples in SE Asia and run your code as-is and publish a paper when you have the sequencers to do the same. (2) Fear of competitors not being able to replicate stochastic ML results; some machine-learning and artificial neural network papers applied to Biology are stochastic. Past authors have been accused of "cherry-picking" the most "optimistic" runs that show positive results, e.g., (https://liorpachter.wordpress.com/2014/02/11/the-network-nonsense-of-manolis-kellis/ https://liorpachter.wordpress.com/2014/02/11/the-network-non...) (3) Focus on science, not tool production; most labs' focus is to produce publications, not open-source software. User base for scientific software is very niche, unlike Web MVC frameworks; making the payoff calculus of packaging and supporting external users not so great. Furthermore, most labs have custom databases making integrating external software, externalizing internal software difficult.

pvnick 11y ago
Just to follow up on your last point, I have found that folks are often happy to send code, but code written in research labs is generally horrendous. I don't think that the "publish or die" culture and a culture of releasing all relevant code is compatible. It takes time to refactor and produce clean code - time that can be spent chasing the next publication.
- kwhitefoot 11y ago
  Then requiring the code to be published and runnable would have a salutary effect on the whole circus.
  - SapphireSun 11y ago
    Not really... I don't see how that would be anything more than adding one more task to a million other tasks. It doesn't change incentives.
- IanCal 11y ago
  I don't see the need for clean code, I just want the code they used. I've repeatedly tried to find the code used to generate various visualisations (which were the entire point of the paper) and then found 1. "It'll be released soon" on a years old page 2. 404'ing personal pages on university sites 3. Finally some code But then the code didn't, and couldn't, generate the images they have. So now I'm not actually sure if the algorithm works as they say. This was also one of the more successful results I've had in trying to find research code outside of ML. I'm sure other disciplines are also good but ML is mostly what I've had to find. There's a part of the experiment which virtually anyone could replicate for a material cost of near 0 and check in detail if they wanted to and yet that seems to be the most poorly explained and shared part of it. You wouldn't get away with just mentioning the protocol you used in the lab without any references or explanations, would you?
ppod 11y ago
(1) You publish the code at the time of publication of the paper, not before. (2) This is a reason to publish the code, not a reason not to publish it. (3) You don't necessarily have to make package it neatly like production software, but some effort is necessary. We just wouldn't accept these kind of excuses for other descriptions of methodology, even though they might apply. There's no reason that code should be an exception.
- cwyers 11y ago
  For 1, I read that as thinking you could get a second paper out of running the same code on another dataset, and not wanting to be scooped on the second paper.
  - IanCal 11y ago
    I understand the thinking but this should sound as non-sensical as > We didn't release our first, second or third papers because we didn't want to be scooped on the fourth and if we publish then someone might build on our work!
    - dagw 11y ago
      We didn't release our first, second or third papers because we didn't want to be scooped on the fourth and if we publish then someone might build on our work! Which happens all the time. If you are working on something big you'll sometimes hold back on publishing parts of your research until you've completed the big thing for just that reason. Then once you've completed the big thing you publish all four papers either together or in quick succession.
  - aiNohY6g 11y ago
    Just running the same code on another dataset is not research, it's production. Without minimal extra scientific efforts, such a work, while valuable, certainly does not deserve a scientific article in a peer review journal.
eli_gottlieb 11y ago
>Fear of competitors not being able to replicate stochastic ML results; some machine-learning and artificial neural network papers applied to Biology are stochastic. Past authors have been accused of "cherry-picking" the most "optimistic" runs that show positive results That sounds like an ML phrasing for "fear of non-reproducibility", which is basically a fear of their research being bad.
IndianAstronaut 11y ago
>Fear of being scooped; This prevents a lot of good research from coming out. We should not just rely on publish or perish. Ifa lab makes a net contribution, they should still be rewarded with grants even if the idea gets "scooped".
Toenex 11y ago
For academic labs you can add 'perceived future commercial value' to that list. In my experience these groups are often run by people with little or no commercial experience and lack the skills to appreciate the difference between a prototype and a commercial solution. Coupled with the pressure from Universities to 'commercialize all the things' this can inhibit attempts to distribute the code freely.
princeb 11y ago
>replicate stochastic ML results (1) use an off the shelf prng (2) provide the data (3) provide the seed

pvnick 11y ago

Just to follow up on your last point, I have found that folks are often happy to send code, but code written in research labs is generally horrendous. I don't think that the "publish or die" culture and a culture of releasing all relevant code is compatible. It takes time to refactor and produce clean code - time that can be spent chasing the next publication.

kwhitefoot 11y ago
Then requiring the code to be published and runnable would have a salutary effect on the whole circus.
- SapphireSun 11y ago
  Not really... I don't see how that would be anything more than adding one more task to a million other tasks. It doesn't change incentives.
IanCal 11y ago
I don't see the need for clean code, I just want the code they used. I've repeatedly tried to find the code used to generate various visualisations (which were the entire point of the paper) and then found 1. "It'll be released soon" on a years old page 2. 404'ing personal pages on university sites 3. Finally some code But then the code didn't, and couldn't, generate the images they have. So now I'm not actually sure if the algorithm works as they say. This was also one of the more successful results I've had in trying to find research code outside of ML. I'm sure other disciplines are also good but ML is mostly what I've had to find. There's a part of the experiment which virtually anyone could replicate for a material cost of near 0 and check in detail if they wanted to and yet that seems to be the most poorly explained and shared part of it. You wouldn't get away with just mentioning the protocol you used in the lab without any references or explanations, would you?

ppod 11y ago

(1) You publish the code at the time of publication of the paper, not before. (2) This is a reason to publish the code, not a reason not to publish it. (3) You don't necessarily have to make package it neatly like production software, but some effort is necessary. We just wouldn't accept these kind of excuses for other descriptions of methodology, even though they might apply. There's no reason that code should be an exception.

cwyers 11y ago
For 1, I read that as thinking you could get a second paper out of running the same code on another dataset, and not wanting to be scooped on the second paper.
- IanCal 11y ago
  I understand the thinking but this should sound as non-sensical as > We didn't release our first, second or third papers because we didn't want to be scooped on the fourth and if we publish then someone might build on our work!
  - dagw 11y ago
    We didn't release our first, second or third papers because we didn't want to be scooped on the fourth and if we publish then someone might build on our work! Which happens all the time. If you are working on something big you'll sometimes hold back on publishing parts of your research until you've completed the big thing for just that reason. Then once you've completed the big thing you publish all four papers either together or in quick succession.
- aiNohY6g 11y ago
  Just running the same code on another dataset is not research, it's production. Without minimal extra scientific efforts, such a work, while valuable, certainly does not deserve a scientific article in a peer review journal.

eli_gottlieb 11y ago

>Fear of competitors not being able to replicate stochastic ML results; some machine-learning and artificial neural network papers applied to Biology are stochastic. Past authors have been accused of "cherry-picking" the most "optimistic" runs that show positive results That sounds like an ML phrasing for "fear of non-reproducibility", which is basically a fear of their research being bad.

IndianAstronaut 11y ago

>Fear of being scooped; This prevents a lot of good research from coming out. We should not just rely on publish or perish. Ifa lab makes a net contribution, they should still be rewarded with grants even if the idea gets "scooped".

Toenex 11y ago

For academic labs you can add 'perceived future commercial value' to that list. In my experience these groups are often run by people with little or no commercial experience and lack the skills to appreciate the difference between a prototype and a commercial solution. Coupled with the pressure from Universities to 'commercialize all the things' this can inhibit attempts to distribute the code freely.

princeb 11y ago

>replicate stochastic ML results (1) use an off the shelf prng (2) provide the data (3) provide the seed