6 ms·

This is such a fundamentally important issue in the sciences. Without the source code, you can not truly do peer review, and you may not be able to replicate so

by whyenot 16y ago

This is such a fundamentally important issue in the sciences. Without the source code, you can not truly do peer review, and you may not be able to replicate someone else's work. With no peer review or replication, you are no longer doing science. I've seen source code for software used to produce results in major publications like Nature that was so poor quality it is surprising it even compiled. You don't generally get tenure for producing well written software, so there needs to be another incentive for scientists to spend the time to write well thought out and well documented code. I think sunlight helps tremendously. Warren DeLano, the author of PyMol, the very popular molecular visualization software, was an early voice about the importance of open source software in the sciences. Unfortunately, he is no longer with us, but it is hearting to see others now taking up the cause. The only way to publish software in a scientifically robust manner is to share source code, and that means publishing via the internet in an open-access/open-source fashion. - Warren DeLano (2005)

jemfinch 16y ago
> With no peer review or replication, you are no longer doing science. Science can be done in isolation. Robinson Crusoe could do science on his small island in the tropics. Without peer review or replication, you're just not doing the particular kind of science that's most beneficial to society: the kind that can be built upon.
- whyenot 16y ago
  I don't agree. The accumulation of a corpus of knowledge is fundamental to the scientific endeavor. Even if you are systematically acquiring knowledge through observation and experimentation, if your work can not be built upon, it's not science.
  - jemfinch 16y ago
    No, the scientific method is fundamental to scientific endeavor. Everything else is just gravy. A man working alone is doing science, he's just not benefiting society as well as he might. Science isn't "pushing the bounds of human knowledge." Science is forming hypotheses and testing them empirically. When kids drop two objects of different weights to see which falls faster, they're doing science just as much as Galileo was, despite the fact that their findings will never get published. What a researcher does doesn't magically become "science" when he publishes. It's science all along. If she does it in a cave for years before she publishes, it's still science for all those years she was in her cave. Children who perform experiments that have been done a million times before are still being scientists. People who perform experiments but never get published are still scientists. Science is about the process, it's not about the result.
    - whyenot 16y ago
      Science existed before the formulation of the scientific method. The accumulation of knowledge has always been fundamental to science going back to Aristotle.
      - jonhendry 16y ago
        Yeah, but a lot of that 'knowledge' was noise, just-so stories, myths, coincidences, best guesses, fanciful explanations, etc.
        jemfinch 16y ago
        There's a reason Aristotle is known primarily as a philosopher and only secondarily as a scientist.
lars512 16y ago
There might not be tenure in it, but there's certainly a lot of citations around for well written tools. The license to use them often requires citation in related work.
jonhendry 16y ago
I'm not sure sharing code is such a great idea. If you use the other guy's code, you might be replicating the other guy's mistakes. And having the code available encourages that. It would be better to describe your methods, so someone else can implement it in their preferred tools. If they still replicate your results, that strikes me as being much stronger than just rerunning the same, possibly buggy, code. I mean, if you're working with monkeys, other scientists can't reuse your monkey. They have to follow the procedures you describe on their own animals. Another issue is that the code might not be very useful to other labs as-is. The code might be for unique, custom-made hardware, or an unusual configuration of equipment, or in an unusual language.
- randomtask 16y ago
  Sharing code doesn't prevent someone else from reimplementing an idea and indeed they should if they rely on results from another publication. If everyone open sources code they wrote during their research then it ought to be easy to tell if someone reimplemented an idea correctly (if their results differ). If they copied the original implementation this should be evident too.
  - almost 16y ago
    But people are people and when there's an easy way...
    - randomtask 16y ago
      I agree, but at least having implementations in the wild should bring more attention to an idea and scrutiny of that code than none?
      - jonhendry 16y ago
        If the same results are obtained using different code (or subjects/animals/chemical batches) does it matter as long as the code is doing the same things? I get the impression that software types are inclined to put way too much significance on the source code. I'm not surprised the author of the linked item in Nature is a software engineer. I wonder, did scientists in the 1930s publish their scratchpads full of calculations?

by whyenot 16y ago

jemfinch 16y ago

> With no peer review or replication, you are no longer doing science. Science can be done in isolation. Robinson Crusoe could do science on his small island in the tropics. Without peer review or replication, you're just not doing the particular kind of science that's most beneficial to society: the kind that can be built upon.

whyenot 16y ago
I don't agree. The accumulation of a corpus of knowledge is fundamental to the scientific endeavor. Even if you are systematically acquiring knowledge through observation and experimentation, if your work can not be built upon, it's not science.
- jemfinch 16y ago
  No, the scientific method is fundamental to scientific endeavor. Everything else is just gravy. A man working alone is doing science, he's just not benefiting society as well as he might. Science isn't "pushing the bounds of human knowledge." Science is forming hypotheses and testing them empirically. When kids drop two objects of different weights to see which falls faster, they're doing science just as much as Galileo was, despite the fact that their findings will never get published. What a researcher does doesn't magically become "science" when he publishes. It's science all along. If she does it in a cave for years before she publishes, it's still science for all those years she was in her cave. Children who perform experiments that have been done a million times before are still being scientists. People who perform experiments but never get published are still scientists. Science is about the process, it's not about the result.
  - whyenot 16y ago
    Science existed before the formulation of the scientific method. The accumulation of knowledge has always been fundamental to science going back to Aristotle.
    - jonhendry 16y ago
      Yeah, but a lot of that 'knowledge' was noise, just-so stories, myths, coincidences, best guesses, fanciful explanations, etc.
      - jemfinch 16y ago
        There's a reason Aristotle is known primarily as a philosopher and only secondarily as a scientist.

lars512 16y ago

There might not be tenure in it, but there's certainly a lot of citations around for well written tools. The license to use them often requires citation in related work.

jonhendry 16y ago

I'm not sure sharing code is such a great idea. If you use the other guy's code, you might be replicating the other guy's mistakes. And having the code available encourages that. It would be better to describe your methods, so someone else can implement it in their preferred tools. If they still replicate your results, that strikes me as being much stronger than just rerunning the same, possibly buggy, code. I mean, if you're working with monkeys, other scientists can't reuse your monkey. They have to follow the procedures you describe on their own animals. Another issue is that the code might not be very useful to other labs as-is. The code might be for unique, custom-made hardware, or an unusual configuration of equipment, or in an unusual language.

randomtask 16y ago
Sharing code doesn't prevent someone else from reimplementing an idea and indeed they should if they rely on results from another publication. If everyone open sources code they wrote during their research then it ought to be easy to tell if someone reimplemented an idea correctly (if their results differ). If they copied the original implementation this should be evident too.
- almost 16y ago
  But people are people and when there's an easy way...
  - randomtask 16y ago
    I agree, but at least having implementations in the wild should bring more attention to an idea and scrutiny of that code than none?
    - jonhendry 16y ago
      If the same results are obtained using different code (or subjects/animals/chemical batches) does it matter as long as the code is doing the same things? I get the impression that software types are inclined to put way too much significance on the source code. I'm not surprised the author of the linked item in Nature is a software engineer. I wonder, did scientists in the 1930s publish their scratchpads full of calculations?