7 ms·

(Disclaimer: my background is in materials physics, and it may be different in other fields. But I doubt it.) Unfortunately there is very little direct incenti

by ajdecon 15y ago

(Disclaimer: my background is in materials physics, and it may be different in other fields. But I doubt it.) Unfortunately there is very little direct incentive for research scientists to write or publish clean, readable code: - There are no direct rewards, in the tenure process or otherwise, for publishing code and having it used by other scientists. Occasionally code which is widely used will add a little to the prestige of an already-eminent scientist, but even then it rarely matters much. - Time spent on anything other than direct research or publication is seen as wasted time, and actively selected against. Especially for young scientists trying to make tenure, also the group most likely to write good code. Many departments actually discourage time spent on teaching, and they're paid to do that. Why would they maintain a codebase? - Most scientific code is written in response to specific problems, usually a body of data or a particular system to be simulated. Because of this, code is often written to the specific problem with little regard for generality, and only rarely re-used. (This leads to lots of wheel re-invention, but it's still done this way.) If you aren't going to re-use your code, why would others? - If by some miracle a researcher produces code which is high-quality and general enough to be used by others, the competitive atmosphere may cause them to want to keep it to themselves. Not as bad a problem in some fields, but I hear biology can be especially bad here. - Most importantly, the software is not the goal. The goal is a better understanding of some natural phenomenon, and a publication. (Or in reverse order...) Why spend more time than absolutely necessary on a single part of the process, especially one that's not in your expertise? And why spend 3x-5x the cost of a research student or postdoc to hire a software developer at competitive rates? I went to grad school in materials science at an R1 institution which was always ranked at 2 or 3 in my field. I wrote a lot of code, mostly image-processing routines for analyzing microscope images. Despite it being essential to understanding my data, the software component of my work was always regarded by my advisor and peers as the least important, most annoying part of the process. Time spent on writing code was seen as wasted, or at best a necessary evil. And it would never be published, so why spend even more time to "make it pretty"? I'm honestly not sure what could be done to improve this. Journals could require that code be submitted with the paper, but I really doubt they'd be motivated to directly enforce any standards, and I have no faith in scientists being embarrassed by bad code. Anything not in the paper itself is usually of secondary importance. (Seriously, if you can, check out how bad the "Supplementary Information" on some papers is.) But even making bad code available could help... I guess. And institutions could try to more directly reward time put into publishing good code, but without the journals on board it may be seen as just another form of "outreach"--i.e., time you should have been in lab. I did publish some code, and exactly two people have contacted me about it. That does make me happy. But many, many more people have contacted me to ask about how I solved some problem in lab, or what I'm working on now that they could connect with. (And are always disappointed when I tell them I left the field, and now work in high-performance computing.) Based on the feedback of my peers... well, on what do you think I should've spent my time?

FrojoS 15y ago
From my experience, even in a field so close to computer science as robotics your analyzes is correct. In my opinion, publishing all code that was used for the paper should be mandatory. Everything else is an obvious violation of the confirmation requirement in the scientific method. But just like with Open Access I have little hope, that this will be adapted soon on a wide scale. If you are a student, I believe, all you can do is get permission to publish your code and do so. Maybe this will hurt tenure but it increases your karma!
shriphani 15y ago
Matt Might has something on this topic. The CRAPL is what he calls it : http://matt.might.net/articles/crapl/ http://matt.might.net/articles/crapl/
JohnnyBrown 15y ago
I'm working in biology now, and a good example of researchers who produce quality, documented code that other people find useful is the Knight group at UC Boulder. They write python with good docs and support, publish the algorithms they come up with in bioinformatics journals, and people cite them all the time. Might be worth thinking about why there are incentives there and not elsewhere.
- roadnottaken 15y ago
  That is an excellent example and a good point. But, for what it's worth, the Knight lab doesn't really do any biology. Most of their biology is done by collaboration with other labs, and the people in the lab are almost entirely programmers or database people. There's nothing wrong with that, but it's more an example of programmers getting into biology than the other way around.
- bluekeybox 15y ago
  Another place where good software work is done is Broad Institute. The reasons are as follows: (a) Broad can hire the best people in bioinformatics, (b) they are a relatively large organization where focusing on the process pays off. Software ultimately is process (i.e. how you do stuff) and small labs often cannot afford to focus on the process and instead try to reach the goal (i.e. publishable results) more or less directly, regardless of the inefficiencies they may encounter. In the past, the model of having many small labs in universities was a great idea. Today things are looking a bit different in some fields because larger labs can afford to do more automation (by hiring programmers instead graduate students).

by ajdecon 15y ago

FrojoS 15y ago

From my experience, even in a field so close to computer science as robotics your analyzes is correct. In my opinion, publishing all code that was used for the paper should be mandatory. Everything else is an obvious violation of the confirmation requirement in the scientific method. But just like with Open Access I have little hope, that this will be adapted soon on a wide scale. If you are a student, I believe, all you can do is get permission to publish your code and do so. Maybe this will hurt tenure but it increases your karma!

shriphani 15y ago

Matt Might has something on this topic. The CRAPL is what he calls it : http://matt.might.net/articles/crapl/ http://matt.might.net/articles/crapl/

JohnnyBrown 15y ago

I'm working in biology now, and a good example of researchers who produce quality, documented code that other people find useful is the Knight group at UC Boulder. They write python with good docs and support, publish the algorithms they come up with in bioinformatics journals, and people cite them all the time. Might be worth thinking about why there are incentives there and not elsewhere.

roadnottaken 15y ago
That is an excellent example and a good point. But, for what it's worth, the Knight lab doesn't really do any biology. Most of their biology is done by collaboration with other labs, and the people in the lab are almost entirely programmers or database people. There's nothing wrong with that, but it's more an example of programmers getting into biology than the other way around.
bluekeybox 15y ago
Another place where good software work is done is Broad Institute. The reasons are as follows: (a) Broad can hire the best people in bioinformatics, (b) they are a relatively large organization where focusing on the process pays off. Software ultimately is process (i.e. how you do stuff) and small labs often cannot afford to focus on the process and instead try to reach the goal (i.e. publishable results) more or less directly, regardless of the inefficiencies they may encounter. In the past, the model of having many small labs in universities was a great idea. Today things are looking a bit different in some fields because larger labs can afford to do more automation (by hiring programmers instead graduate students).