5 ms·

Finally. Finally a discussion about this.

by maurycy 17y ago

timr 17y ago
Enough with the false melodrama, please. Aside from the fact that your comment is content-free and inane, scientists have been discussing this subject since computer simulation first became a part of science. A lot of scientists do share their code (I'm one of them, and I believe in sharing code). But there are good arguments on the other side. Among them: 1) Papers describe methods in enough detail to reproduce them. If they don't, there's a serious problem. 2) Independent lines of verification. If simulation code becomes a reference, it's inevitable that the same bugs/bad assumptions will contaminate an entire field. Independent re-implementation of the same algorithms is a strong hedge against this phenomenon (even if it means that there are more bugs overall). 3) Money. A lot of scientists fund their research in part through licensing of implementations of their algorithms. I don't like it, but until someone gets around to repealing Bayh-Dole (a real scientific travesty, IMO), this is going to continue to be a problem. In short, what you really meant to say was that finally someone wrote a newspaper article about this subject. It's not a new discussion.
- lolcraft 17y ago
  I'll add another argument for not releasing scientific code, specially of a matter so polarizing as climate change: programmers aren't scientists. We aren't familiar with neither the field's literature nor the science being used. For example, the famous comment from the CRU code said: ; Plots 24 yearly maps of calibrated (PCR-infilled or not) MXD reconstructions ; of growing season temperatures. Uses “corrected” MXD – but shouldn’t usually ; plot past 1960 because these will be artificially adjusted to look closer to ; the real temperatures. Do you expect the average programmer to take a 2 year course in climate science to truly understand what MXD means instead of instantly running around screaming "wolf!"? If you did so, be disappointed (a careful reading of Google results for "CRU code" might be enlightening).
  - jgrahamc 17y ago
    That seems like a bogus reason. Some people who had already decided that climate change was fraud decided to read comments in the code and interpret them to fit their worldview. Other people read the CRU code and made intelligent comments about the code.
    - matthewmarkus 17y ago
      Hear, hear! I did a whole series of posts analyzing a skeptic's claims: http://matthewmarkus.posterous.com/raw-darwin-airport-temperature-data-with-r http://matthewmarkus.posterous.com/raw-darwin-airport-temper... http://matthewmarkus.posterous.com/darwin-airport-temperature-data-revisited http://matthewmarkus.posterous.com/darwin-airport-temperatur... http://matthewmarkus.posterous.com/darwin-airport-temperature-data-in-context http://matthewmarkus.posterous.com/darwin-airport-temperatur... I found at least one error/discrepancy in each side's output. Of course, neither side released any code. The disclosure of methods is a prerequisite for repeatable experiments, the cornerstone of science.
- DaniFong 17y ago
  Closed academic publishing is intellectually bankrupt, and is probably one of the greatest problems effecting research today. People don't share code, and put a paywall between themselves and the public. There are open journals, but they are rarely as prestigious, and so are not as valuable to those seeking tenure. These academics put tenure before fruitful scientific discussion.
  - Retric 17y ago
    A respected journal needs to be able to pay for someone to review a paper before publication. I suspect that setting up a foundation to do this for free would be a great charity, but without some sort of backing you can't create high quality.
    - michael_nielsen 17y ago
      In nearly all fields, referees of scientific papers are not paid. Referees of scientific books may be paid a small honorarium, but, compared with consulting, it's a pittance. Background experience: I refereed somewhat over 100 papers and perhaps a dozen or so books during my career as a physicist. My work now overlaps with the scientific publishing industry more broadly.
      - Retric 17y ago
        It is my understanding that referees are sometimes paid, but editors rarely work for free. (Using http://en.wikipedia.org/wiki/Peer_review http://en.wikipedia.org/wiki/Peer_review version of editors.) It's a lot of work to find out who is best to referee a given work and to track down conflicts of interest etc.
        michael_nielsen 17y ago
        With editors the issue of payment depends on the journal. Some journals employ a staff of professional editors. Others recruit tenured scientists who do the editorial work for an honorarium that seems tiny, considering the amount of work involved. I don't know which model is more widely used - I can think offhand of many journals of both types, but don't recall ever having seen statistics.
- paulkirk 17y ago
  Those are bad reasons on all counts: 1) Code is notoriosly difficult to describe in a natural language, so it's unlikely papers contain enough detail to replicate a complex program. 2) People can still reimplement the model even when it's available. In fact it's going to be more likely that non-scientists join the convesation (which will improve the abysmall public perception of science, especially climate science). 3) Making money is great for an inventor and/or startup founder. You can still sell implementations, but they can't be trusted until they have been run on a large number of independent computers. Until then, it's not science.

Finally. Finally a discussion about this.

by maurycy 17y ago

Finally. Finally a discussion about this.

timr 17y ago

Enough with the false melodrama, please. Aside from the fact that your comment is content-free and inane, scientists have been discussing this subject since computer simulation first became a part of science. A lot of scientists do share their code (I'm one of them, and I believe in sharing code). But there are good arguments on the other side. Among them: 1) Papers describe methods in enough detail to reproduce them. If they don't, there's a serious problem. 2) Independent lines of verification. If simulation code becomes a reference, it's inevitable that the same bugs/bad assumptions will contaminate an entire field. Independent re-implementation of the same algorithms is a strong hedge against this phenomenon (even if it means that there are more bugs overall). 3) Money. A lot of scientists fund their research in part through licensing of implementations of their algorithms. I don't like it, but until someone gets around to repealing Bayh-Dole (a real scientific travesty, IMO), this is going to continue to be a problem. In short, what you really meant to say was that finally someone wrote a newspaper article about this subject. It's not a new discussion.

lolcraft 17y ago
I'll add another argument for not releasing scientific code, specially of a matter so polarizing as climate change: programmers aren't scientists. We aren't familiar with neither the field's literature nor the science being used. For example, the famous comment from the CRU code said: ; Plots 24 yearly maps of calibrated (PCR-infilled or not) MXD reconstructions ; of growing season temperatures. Uses “corrected” MXD – but shouldn’t usually ; plot past 1960 because these will be artificially adjusted to look closer to ; the real temperatures. Do you expect the average programmer to take a 2 year course in climate science to truly understand what MXD means instead of instantly running around screaming "wolf!"? If you did so, be disappointed (a careful reading of Google results for "CRU code" might be enlightening).
- jgrahamc 17y ago
  That seems like a bogus reason. Some people who had already decided that climate change was fraud decided to read comments in the code and interpret them to fit their worldview. Other people read the CRU code and made intelligent comments about the code.
  - matthewmarkus 17y ago
    Hear, hear! I did a whole series of posts analyzing a skeptic's claims: http://matthewmarkus.posterous.com/raw-darwin-airport-temperature-data-with-r http://matthewmarkus.posterous.com/raw-darwin-airport-temper... http://matthewmarkus.posterous.com/darwin-airport-temperature-data-revisited http://matthewmarkus.posterous.com/darwin-airport-temperatur... http://matthewmarkus.posterous.com/darwin-airport-temperature-data-in-context http://matthewmarkus.posterous.com/darwin-airport-temperatur... I found at least one error/discrepancy in each side's output. Of course, neither side released any code. The disclosure of methods is a prerequisite for repeatable experiments, the cornerstone of science.
DaniFong 17y ago
Closed academic publishing is intellectually bankrupt, and is probably one of the greatest problems effecting research today. People don't share code, and put a paywall between themselves and the public. There are open journals, but they are rarely as prestigious, and so are not as valuable to those seeking tenure. These academics put tenure before fruitful scientific discussion.
- Retric 17y ago
  A respected journal needs to be able to pay for someone to review a paper before publication. I suspect that setting up a foundation to do this for free would be a great charity, but without some sort of backing you can't create high quality.
  - michael_nielsen 17y ago
    In nearly all fields, referees of scientific papers are not paid. Referees of scientific books may be paid a small honorarium, but, compared with consulting, it's a pittance. Background experience: I refereed somewhat over 100 papers and perhaps a dozen or so books during my career as a physicist. My work now overlaps with the scientific publishing industry more broadly.
    - Retric 17y ago
      It is my understanding that referees are sometimes paid, but editors rarely work for free. (Using http://en.wikipedia.org/wiki/Peer_review http://en.wikipedia.org/wiki/Peer_review version of editors.) It's a lot of work to find out who is best to referee a given work and to track down conflicts of interest etc.
      - michael_nielsen 17y ago
        With editors the issue of payment depends on the journal. Some journals employ a staff of professional editors. Others recruit tenured scientists who do the editorial work for an honorarium that seems tiny, considering the amount of work involved. I don't know which model is more widely used - I can think offhand of many journals of both types, but don't recall ever having seen statistics.
paulkirk 17y ago
Those are bad reasons on all counts: 1) Code is notoriosly difficult to describe in a natural language, so it's unlikely papers contain enough detail to replicate a complex program. 2) People can still reimplement the model even when it's available. In fact it's going to be more likely that non-scientists join the convesation (which will improve the abysmall public perception of science, especially climate science). 3) Making money is great for an inventor and/or startup founder. You can still sell implementations, but they can't be trusted until they have been run on a large number of independent computers. Until then, it's not science.