6 ms·

I used to call the scientific software that I was writing, "paper-ware". You aren't building a system for other users, you aren't really doing anything other t

I used to call the scientific software that I was writing, "paper-ware". You aren't building a system for other users, you aren't really doing anything other than one-off analysis to create charts, which will be explained in a paper. Things have changed somewhat since the early 2000's, but the concept remains the same. Nowadays, for interesting or controversial results other scientists want to be able to verify your results. However, that is usually more related to your data and how you processed it, rather than your software algorithms (which should be explained in the paper, and can be recreated from that). So do these systems need to have reams of documentation? Probably not. However, if you leave the system for two years and come back to work on it, or figure out how it used to work, then you best have enough commenting with a thorough readme about some of the decisions you made and why. It's more analogous to scripting rather than software engineering.

shardling 14y ago
On the other hand, if you put good software out that people use, it counts as a citation. The most cited prof in the dept I graduated from maintained a widely used program for astronomical simulations.
- Xcelerate 14y ago
  Seriously? That's intriguing... If you wrote a very popular scientific library, how much would that impact your career compared to, for example, a Nature publication? (I ask this because, for me, I think the former is much more likely than the latter.)
  - epistasis 14y ago
    They serve somewhat different purposes, but an influential library will have far far more citations. At least in bioinformatics, that's the only way to get a large number of citations, because at their core bioinformaticians are tool builders! For example, samtools has ~1400 citations [1]. The original BLAST paper is around ~44k citations [2]. However, there are a ton of bioinformatics libraries released every year, and almost none of them gain any traction. Nature publications are far more frequent than important new libraries, and you need more political clout to get the library popular than you need to get a Nature paper. Really, you need to gauge usefulness and interest of your library before you devote a ton of time to it. It's a lot like a startup's product. [1] http://scholar.google.com/scholar?q=samtools http://scholar.google.com/scholar?q=samtools [2] http://scholar.google.com/scholar?q=BLAST http://scholar.google.com/scholar?q=BLAST
    - michaelbarton 14y ago
      Thank you. This is the point I tried to make when I wrote the original post. What's the point of spending extra time creating documentation and examples if, on average, no one is going to use or try your software. Perhaps it's just best to create a small publication for the software then push it out the door.
    - mgillett 14y ago
      I think this is more a comment on how the system is broken. Researchers should be notified of new libraries in their area. At the very least, they should be able to consult a single site that everyone uploads their code to (think Github for science with more emphasis on exploration). Academic journals are not the only channels carrying useful information.
      - epistasis 14y ago
        It's not the discoverability of the libraries that's the problem, it's that the utility of these libraries is generally not that great for anyone except the authors. One common type of library handles data transformation, normalization, and maybe even workflows. These abound. But they are rarely useful in other people's hands, because to extend them and actually get any work done, you need to spend as much time learning them as it would take to write it from scratch. And the advantage of writing it from scratch is that you know it intimately, and all of its assumptions and flaws, which you don't know about somebody else's code, even if it's extremely well documented. Take something like Taverna [1], which is probably very useful to some people, and had been recommended enthusiastically to me by many people, but after spending three hours reading documents and searching the web, I could not get it to do what I needed to do, so I wrote a simple one-off bash script that interfaced with our cluster system. Alternatively I could try to hack in loops, but that's going to take me 10x as long, will require me to interact with many other people who obviously don't understand my problem since they did not consider it a fundamental need, and may not even be accepted back into the mainline, at which point I'm off on my own fork and lose the benefit of using a common code base. Waiting 1-10 hours to hear back from the dev mailing list is unacceptable when you're trying to get work done. Is it more important to get the result, or to use other people's code? Reinventing the wheel is a minor sin compared to not getting results. [1] http://www.taverna.org.uk http://www.taverna.org.uk
  - mturmon 14y ago
    Within machine learning, Peter Cheeseman developed Autoclass, a very popular clustering program which is probably his most highly cited work. Kevin Murphy used to be very well known, when he was a postdoc, for his Bayes net toolbox for Matlab. The author of svmlight is another example; the package was developed during his phd. I can't recall for sure, but I think FFTW started out as a student's project. The key is to find a modular component that is difficult or novel, and at the same time broadly applicable and hopefully extensible.

shardling 14y ago

On the other hand, if you put good software out that people use, it counts as a citation. The most cited prof in the dept I graduated from maintained a widely used program for astronomical simulations.

Xcelerate 14y ago
Seriously? That's intriguing... If you wrote a very popular scientific library, how much would that impact your career compared to, for example, a Nature publication? (I ask this because, for me, I think the former is much more likely than the latter.)
- epistasis 14y ago
  They serve somewhat different purposes, but an influential library will have far far more citations. At least in bioinformatics, that's the only way to get a large number of citations, because at their core bioinformaticians are tool builders! For example, samtools has ~1400 citations [1]. The original BLAST paper is around ~44k citations [2]. However, there are a ton of bioinformatics libraries released every year, and almost none of them gain any traction. Nature publications are far more frequent than important new libraries, and you need more political clout to get the library popular than you need to get a Nature paper. Really, you need to gauge usefulness and interest of your library before you devote a ton of time to it. It's a lot like a startup's product. [1] http://scholar.google.com/scholar?q=samtools http://scholar.google.com/scholar?q=samtools [2] http://scholar.google.com/scholar?q=BLAST http://scholar.google.com/scholar?q=BLAST
  - michaelbarton 14y ago
    Thank you. This is the point I tried to make when I wrote the original post. What's the point of spending extra time creating documentation and examples if, on average, no one is going to use or try your software. Perhaps it's just best to create a small publication for the software then push it out the door.
  - mgillett 14y ago
    I think this is more a comment on how the system is broken. Researchers should be notified of new libraries in their area. At the very least, they should be able to consult a single site that everyone uploads their code to (think Github for science with more emphasis on exploration). Academic journals are not the only channels carrying useful information.
    - epistasis 14y ago
      It's not the discoverability of the libraries that's the problem, it's that the utility of these libraries is generally not that great for anyone except the authors. One common type of library handles data transformation, normalization, and maybe even workflows. These abound. But they are rarely useful in other people's hands, because to extend them and actually get any work done, you need to spend as much time learning them as it would take to write it from scratch. And the advantage of writing it from scratch is that you know it intimately, and all of its assumptions and flaws, which you don't know about somebody else's code, even if it's extremely well documented. Take something like Taverna [1], which is probably very useful to some people, and had been recommended enthusiastically to me by many people, but after spending three hours reading documents and searching the web, I could not get it to do what I needed to do, so I wrote a simple one-off bash script that interfaced with our cluster system. Alternatively I could try to hack in loops, but that's going to take me 10x as long, will require me to interact with many other people who obviously don't understand my problem since they did not consider it a fundamental need, and may not even be accepted back into the mainline, at which point I'm off on my own fork and lose the benefit of using a common code base. Waiting 1-10 hours to hear back from the dev mailing list is unacceptable when you're trying to get work done. Is it more important to get the result, or to use other people's code? Reinventing the wheel is a minor sin compared to not getting results. [1] http://www.taverna.org.uk http://www.taverna.org.uk
- mturmon 14y ago
  Within machine learning, Peter Cheeseman developed Autoclass, a very popular clustering program which is probably his most highly cited work. Kevin Murphy used to be very well known, when he was a postdoc, for his Bayes net toolbox for Matlab. The author of svmlight is another example; the package was developed during his phd. I can't recall for sure, but I think FFTW started out as a student's project. The key is to find a modular component that is difficult or novel, and at the same time broadly applicable and hopefully extensible.