5 ms·

I'm a computational biologist. If I read a paper where the results of an analysis are presented, but the code used is not, that analysis is worthless. Thankfull

by Blahah 14y ago

I'm a computational biologist. If I read a paper where the results of an analysis are presented, but the code used is not, that analysis is worthless. Thankfully, in computational biology & bioinformatics at least, open source is widespread and growing. Every programmer knows that every programmer makes mistakes. Lots of them. Mistakes that can completely change the outcome of an analysis. And if the results of your analysis are to be taken seriously, people have to be able to trust them, which means they have to be able to check your working. Not publishing your code is anti-science, selfish, and IMHO should be disallowed in the literature.

anon_barcode 14y ago
In astronomy we are still fighting for public data. Verification of results comes from another researcher collecting his own data and running his own analysis. Out of curiosity, do you work for industry or academia?
- Blahah 14y ago
  That's an extremely inefficient and costly way to validate a result! It would be rendered unnecessary in cases where just looking at the code invalidated the result. I'm in academia (see my profile for details).
  - J_Thomas 14y ago
    It's good and useful for independent researchers to collect their own data and analyse it. That's a check on the data collection methods. It's also useful for others to check the analysis of trusted data. If they do it with their own methods and get a different result, then it's time to actually compare the methods. It's a good thing for the original researchers to release their code too. But looking for bugs in the code is no substitute for doing independent analysis and independent data collection. It's an important supplement, but it isn't enough.
    - Blahah 14y ago
      I agree with your points. The saving in the case of released source code is that, if the original analysis is flawed in such a way that there's no need to repeat it because it's wrong, then a large amount of expense and effort can be saved. Of course, if the original analysis holds up given the data and code, independent replication is a necessary thing.
  - dalke 14y ago
    Sometimes validation can be done without knowing the actual algorithm or being able to rerun it. For example, if a paper publishes a multiple sequence alignment of 100 sequences, then all you need to do is verify that it's at least as good as what other MSA programs generate. You don't need to be able to rerun the program. Indeed, sometimes you can't. Someone could have manually aligned the sequence, and that process isn't reproducible. You see this as well with fold prediction software, where it can be easy to show that a predicted fold is likely correct (ie, matches experimental observations), and where you don't necessarily care about the method used to get that fold prediction. A genetic algorithm might be very sensitive to the compute environment. For example, the order of float operations generated by two different compiler settings, or by network traffic timings in a distributed system. This can lead to different minima, where the overall effectiveness is the same but the actual configuration is different. The GA search doesn't need to be reproducible; it's only the final effectiveness which is important. For those cases, I don't see the need for access to the underlying software in order to validate the result.
    - Blahah 14y ago
      The problem with the MSA question is that there is no objective measure of goodness. Most people compare them to a subjective 'gold standard', which is a manual alignment. Black box comparisons are not progressive: the only non-subjective advances can come from incorporating the results of sequence evolution experiments into the algorithms, and then the algorithm needs to be trusted, for which it needs to be open to scrutiny. I agree about the genetic algorithm.
new299 14y ago
If the paper is based on the analysis of sequence data, it's almost certainly been processed by closed or "shared source" primary data analysis tools. That's true of Illumina/Roche/ABI. Biotech vendors have quite a long way to go in this regard.
dalke 14y ago
"Not publishing your code is anti-science" ... "open source is widespread and growing" I've noticed that few people realize that "publishing your code" and "open source" are two overlapping issues, but they are not the same and the needs of one field are sometimes opposed to the other field. Open source necessarily implies that the people who receive the software have the right to modify the software, redistribute it with or without changes, and be able to do so for a charge. No one has been able to tell me why software for scientific publications requires the third of those abilities. Everyone is agreed that access to the code and the ability to modify it makes it much easier to review and understand what it does. I can also understand the need to share modifications with collaborators, in order to help carry out the analysis. But I don't see why published software can't have a prohibition on charging a fee for using modified versions of the software, or services rendered which use the software. My first question for you, then, is: do you need the ability to commercialize someone else's software in order to provide good scientific review of their publication? More specifically, what sorts of review would that prohibition eliminate? In the other direction, I have scientific software which I sell for about $30K. (This is not hypothetical - I really do have this). Customers get the source code under the BSD license. This falls into every standard definition of "free software" and "open source" software. There's even an essay (at http://www.gnu.org/philosophy/selling.html http://www.gnu.org/philosophy/selling.html ) encouraging people to sell free software. If I publish a paper which uses the software, then am I obligated to give my peers access to the source code for no fee? Or can I publish the paper, say it's available under a BSD license, and charge $30K for access to it? So my second question is, are there limits on what I can charge people in order to get access to my open source software, which was used for a paper? If so, what are they, and what is the ethical basis for that judgment? (For example, should it be "fair, reasonable and non-discriminatory"? Can it cover development costs? Distribution costs? Web site development costs?)
- Blahah 14y ago
  To clarify, I'm saying that publishing the source code so that others can run and modify it is the crucial thing. I'm not saying that not open sourcing your code is anti-science, but not making it available for scrutiny is. So to answer your first question, I don't think the right to sell someone else's code is related to the scientific process. I can't think of a way it affects the progress of science to use (or not use) commercialisation-friendly licenses, except in some indirect economic ways I don't know enough about to predict. As to the second question, I think if you tried to charge reviewers for access to your code, your paper would never be accepted at a journal. Science is largely public funded. When public funds are used for research, the fruits of that research should be made public. That includes your code.

by Blahah 14y ago

anon_barcode 14y ago

In astronomy we are still fighting for public data. Verification of results comes from another researcher collecting his own data and running his own analysis. Out of curiosity, do you work for industry or academia?

Blahah 14y ago
That's an extremely inefficient and costly way to validate a result! It would be rendered unnecessary in cases where just looking at the code invalidated the result. I'm in academia (see my profile for details).
- J_Thomas 14y ago
  It's good and useful for independent researchers to collect their own data and analyse it. That's a check on the data collection methods. It's also useful for others to check the analysis of trusted data. If they do it with their own methods and get a different result, then it's time to actually compare the methods. It's a good thing for the original researchers to release their code too. But looking for bugs in the code is no substitute for doing independent analysis and independent data collection. It's an important supplement, but it isn't enough.
  - Blahah 14y ago
    I agree with your points. The saving in the case of released source code is that, if the original analysis is flawed in such a way that there's no need to repeat it because it's wrong, then a large amount of expense and effort can be saved. Of course, if the original analysis holds up given the data and code, independent replication is a necessary thing.
- dalke 14y ago
  Sometimes validation can be done without knowing the actual algorithm or being able to rerun it. For example, if a paper publishes a multiple sequence alignment of 100 sequences, then all you need to do is verify that it's at least as good as what other MSA programs generate. You don't need to be able to rerun the program. Indeed, sometimes you can't. Someone could have manually aligned the sequence, and that process isn't reproducible. You see this as well with fold prediction software, where it can be easy to show that a predicted fold is likely correct (ie, matches experimental observations), and where you don't necessarily care about the method used to get that fold prediction. A genetic algorithm might be very sensitive to the compute environment. For example, the order of float operations generated by two different compiler settings, or by network traffic timings in a distributed system. This can lead to different minima, where the overall effectiveness is the same but the actual configuration is different. The GA search doesn't need to be reproducible; it's only the final effectiveness which is important. For those cases, I don't see the need for access to the underlying software in order to validate the result.
  - Blahah 14y ago
    The problem with the MSA question is that there is no objective measure of goodness. Most people compare them to a subjective 'gold standard', which is a manual alignment. Black box comparisons are not progressive: the only non-subjective advances can come from incorporating the results of sequence evolution experiments into the algorithms, and then the algorithm needs to be trusted, for which it needs to be open to scrutiny. I agree about the genetic algorithm.

new299 14y ago

If the paper is based on the analysis of sequence data, it's almost certainly been processed by closed or "shared source" primary data analysis tools. That's true of Illumina/Roche/ABI. Biotech vendors have quite a long way to go in this regard.

dalke 14y ago

"Not publishing your code is anti-science" ... "open source is widespread and growing" I've noticed that few people realize that "publishing your code" and "open source" are two overlapping issues, but they are not the same and the needs of one field are sometimes opposed to the other field. Open source necessarily implies that the people who receive the software have the right to modify the software, redistribute it with or without changes, and be able to do so for a charge. No one has been able to tell me why software for scientific publications requires the third of those abilities. Everyone is agreed that access to the code and the ability to modify it makes it much easier to review and understand what it does. I can also understand the need to share modifications with collaborators, in order to help carry out the analysis. But I don't see why published software can't have a prohibition on charging a fee for using modified versions of the software, or services rendered which use the software. My first question for you, then, is: do you need the ability to commercialize someone else's software in order to provide good scientific review of their publication? More specifically, what sorts of review would that prohibition eliminate? In the other direction, I have scientific software which I sell for about $30K. (This is not hypothetical - I really do have this). Customers get the source code under the BSD license. This falls into every standard definition of "free software" and "open source" software. There's even an essay (at http://www.gnu.org/philosophy/selling.html http://www.gnu.org/philosophy/selling.html ) encouraging people to sell free software. If I publish a paper which uses the software, then am I obligated to give my peers access to the source code for no fee? Or can I publish the paper, say it's available under a BSD license, and charge $30K for access to it? So my second question is, are there limits on what I can charge people in order to get access to my open source software, which was used for a paper? If so, what are they, and what is the ethical basis for that judgment? (For example, should it be "fair, reasonable and non-discriminatory"? Can it cover development costs? Distribution costs? Web site development costs?)

Blahah 14y ago
To clarify, I'm saying that publishing the source code so that others can run and modify it is the crucial thing. I'm not saying that not open sourcing your code is anti-science, but not making it available for scrutiny is. So to answer your first question, I don't think the right to sell someone else's code is related to the scientific process. I can't think of a way it affects the progress of science to use (or not use) commercialisation-friendly licenses, except in some indirect economic ways I don't know enough about to predict. As to the second question, I think if you tried to charge reviewers for access to your code, your paper would never be accepted at a journal. Science is largely public funded. When public funds are used for research, the fruits of that research should be made public. That includes your code.