7 ms·

How a coding error fueled a dispute between two condensed matter theorists

southern_cross 8y ago
This article harkens back to discussions of the "reproducibility crisis" in science, as discussed extensively here just recently (see link below). Where, in this case, not coughing up the code used in the simulations in a timely manner led to an apparently unnecessary multi-year dispute. https://news.ycombinator.com/item?id=17702380 https://news.ycombinator.com/item?id=17702380
- BeetleB 8y ago
  I'm glad this is getting to the fore. Depending on which forum I'm in, I'm heavily downvoted when I talk about reproducibility problems in hard sciences - particularly physics. My thesis was in condensed matter, and (in those days), no one published code. There were a lot of pointless arguments like the one in this submission (although usually not as heated) - all because professors were against sharing their code. It was routine to hear researchers discuss whether they believed a particular paper. Why should belief be a factor? Simulation code contains a lot of numerical tricks and hacks - and the papers never discuss them.[1] So reproducing the results of a paper were mostly pointless, and the general practice was "Let's see if we can reproduce their findings using our technique". If you got somewhat similar results, you'd publish. If you didn't, you'd sometimes publish (remember, there's a bias against negative results). [1]In fact, one reviewer even insisted that those details be removed because the journal was about publishing interesting new findings in science - not interesting techniques to make the computation more amenable.
fwdpropaganda 8y ago
Having been in the academia I can tell you that from my experience there's two main reasons why scientists are often reluctant to share their code: A) embarassement at having other people see how bad it is. Not necessarily wrong, but bad in other ways. B) wanting to keep an advantage by continuing to publish on top of the work already done, whereas anyone else wanting to get in on the same idea would have to first re-build the original work.
- ApostleMatthew 8y ago
  In my case, it is very much column A.
  - fwdpropaganda 8y ago
    It's the most common one, that's why I put it first :D
  - ur-whale 8y ago
    If you're embarrassed by your research code, how do you know it works, and how do you know what you're going to publish isn't built on quicksand?
    - rrmm 8y ago
      Usually testing. Just because code works flawlessly doesn't mean it isn't embarrassing; and just because you are proud of the code doesn't mean it's bug free.
    - ApostleMatthew 8y ago
      Embarrassed by its cobbled-togetherness. Not in whether or not it works.
      - kobigurk 8y ago
        I understand the embarrassment, I am sometimes embarrassed as well looking at quickly-put-together code I do appreciate papers that give you either a reference implementation or at least enough details on how it was done. I spent too many hours on papers that didn't trying to reproduce their claimed results...
      - ur-whale 8y ago
        Not really answering the question, which was "how do you know it works?" I would contend that if you have proven to yourself that your code works and are therefore capable of proving it to other folks (via e.g. solid testing), you should not be ashamed of the spit and glue. It is research code, we all know what that looks like. But if, on the other hands you haven't proven to yourself it works, then it's definitely something to be ashamed of - scientifically speaking.
ApostleMatthew 8y ago
My PhD adviser was a good friend of Chandler's, and a lot of my doctoral work was based on previous work done in his lab. I'm honestly really surprised one of his students/post-docs made a mistake of this magnitude -- generating a velocity distribution which does not follow the Boltzmann distribution is not something I'd expect to be a result of code coming from his lab. But we all make mistakes, and this really just lends more weight to the idea that all code used in producing published results should be posted along with the article itself, just like data sets often are (at least from studies with government funding).
- ur-whale 8y ago
  Refusing to share their code falls under the "we all make mistakes" umbrella you think?
  - whateveryou381 8y ago
    Sharing code is not always the answer. Independent thinking implies independence, in code and more abstractly. The real 'mistake' is allowing the drama to escalate to the point that it is toxic. People who are truly interested don't care about who is right and who is wrong. When disagreements like this come up, having common and good test cases are probably the most important (and is indeed the way in which the problem, generation of non-Boltzman behavior, was found). In the end ST2 itself is flawed, and Princeton admitted that the discussion of water was not significantly advanced through this drama. Is it worth it? Understanding the argument and its importance should be the focus.
    - YorkshireSeason 8y ago
      Sharing code is not always the answer Why let the perfect be the enemy of the good? Sharing code is not the full answer, especially in case of Monte-Carlo simulations in physics, because that kind of algorithm is hard to test: What is the testing oracle? What is the specification? But sharing code is part of the answer. Setting up a culture where it is unacceptable to submit a paper without open-sourcing the code and suitable testing (for simple edge ases), and suitable scripts that make reproducing the software simulations easy, is good scientific 'hygiene'. See for example [1, 2] for efforts towards reproducible software submissions in computer science. Reproduciblility is the very essence of the scientific method. [1] http://evaluate.inf.usi.ch/artifacts http://evaluate.inf.usi.ch/artifacts [2] http://www.artifact-eval.org/ http://www.artifact-eval.org/
ur-whale 8y ago
Modern science journals, in the same way they don't accept articles without proper reference to previous work, should not accept research articles unless there's a link to the source code in the article.
bigiain 8y ago
> “I had and was very willing to share the code,” he says. What he didn’t have, he says, was the time or personnel to prepare the code in a form that could be useful to an outsider. That's what this license is for: http://matt.might.net/articles/crapl/ http://matt.might.net/articles/crapl/
- itronitron 8y ago
  Considering that a paper is run through a subjective and highly critical peer-review I can see why authors would be reluctant to share their code.
  - bigiain 8y ago
    It's an interesting observation then that the people doing the "highly critical peer review" might promote papers with no source code showing how the calculations were done over papers with messy-but-available source code. I know some fields intentionally/explicitly hide their working ("Law and sausage are two things you do not want to see being made."), but I wouldn't have thought academia would subscribe to that... There's a lot to be said for the IETF's policy of "rough consensus and running code".
    - itronitron 8y ago
      The paper should show how the calculations can be made, along with the results of that calculation, preferably on a publicly available dataset. A common strategy is to describe the calculation in highly convoluted or specialized language (mathematical formulas) in order to reduce the number of people that may try to reproduce the work. Having reviewed papers on multiple occasions, I wouldn't spend much if any time looking at the actual source code if it is available. Every paper submission is rushed because it needs to be ahead of what anyone else is doing (at least in CS and related fields).
      - bigiain 8y ago
        Do you see this as an inevitable "that's the way it is" part of how the peer review process works? Or as a flaw in the process? Or something in between. (A deeply cynical and suspicious part of me suspects this is because the prime motivator for "peer reviewed journals" is to make Elsiver richer, and academic knowledge sharing and reputation building is just a side effect of that...)

This article harkens back to discussions of the "reproducibility crisis" in science, as discussed extensively here just recently (see link below). Where, in this case, not coughing up the code used in the simulations in a timely manner led to an apparently unnecessary multi-year dispute. https://news.ycombinator.com/item?id=17702380 https://news.ycombinator.com/item?id=17702380

BeetleB 8y ago
I'm glad this is getting to the fore. Depending on which forum I'm in, I'm heavily downvoted when I talk about reproducibility problems in hard sciences - particularly physics. My thesis was in condensed matter, and (in those days), no one published code. There were a lot of pointless arguments like the one in this submission (although usually not as heated) - all because professors were against sharing their code. It was routine to hear researchers discuss whether they believed a particular paper. Why should belief be a factor? Simulation code contains a lot of numerical tricks and hacks - and the papers never discuss them.[1] So reproducing the results of a paper were mostly pointless, and the general practice was "Let's see if we can reproduce their findings using our technique". If you got somewhat similar results, you'd publish. If you didn't, you'd sometimes publish (remember, there's a bias against negative results). [1]In fact, one reviewer even insisted that those details be removed because the journal was about publishing interesting new findings in science - not interesting techniques to make the computation more amenable.

fwdpropaganda 8y ago

Having been in the academia I can tell you that from my experience there's two main reasons why scientists are often reluctant to share their code: A) embarassement at having other people see how bad it is. Not necessarily wrong, but bad in other ways. B) wanting to keep an advantage by continuing to publish on top of the work already done, whereas anyone else wanting to get in on the same idea would have to first re-build the original work.

ApostleMatthew 8y ago
In my case, it is very much column A.
- fwdpropaganda 8y ago
  It's the most common one, that's why I put it first :D
- ur-whale 8y ago
  If you're embarrassed by your research code, how do you know it works, and how do you know what you're going to publish isn't built on quicksand?
  - rrmm 8y ago
    Usually testing. Just because code works flawlessly doesn't mean it isn't embarrassing; and just because you are proud of the code doesn't mean it's bug free.
  - ApostleMatthew 8y ago
    Embarrassed by its cobbled-togetherness. Not in whether or not it works.
    - kobigurk 8y ago
      I understand the embarrassment, I am sometimes embarrassed as well looking at quickly-put-together code I do appreciate papers that give you either a reference implementation or at least enough details on how it was done. I spent too many hours on papers that didn't trying to reproduce their claimed results...
    - ur-whale 8y ago
      Not really answering the question, which was "how do you know it works?" I would contend that if you have proven to yourself that your code works and are therefore capable of proving it to other folks (via e.g. solid testing), you should not be ashamed of the spit and glue. It is research code, we all know what that looks like. But if, on the other hands you haven't proven to yourself it works, then it's definitely something to be ashamed of - scientifically speaking.

ApostleMatthew 8y ago

My PhD adviser was a good friend of Chandler's, and a lot of my doctoral work was based on previous work done in his lab. I'm honestly really surprised one of his students/post-docs made a mistake of this magnitude -- generating a velocity distribution which does not follow the Boltzmann distribution is not something I'd expect to be a result of code coming from his lab. But we all make mistakes, and this really just lends more weight to the idea that all code used in producing published results should be posted along with the article itself, just like data sets often are (at least from studies with government funding).

ur-whale 8y ago
Refusing to share their code falls under the "we all make mistakes" umbrella you think?
- whateveryou381 8y ago
  Sharing code is not always the answer. Independent thinking implies independence, in code and more abstractly. The real 'mistake' is allowing the drama to escalate to the point that it is toxic. People who are truly interested don't care about who is right and who is wrong. When disagreements like this come up, having common and good test cases are probably the most important (and is indeed the way in which the problem, generation of non-Boltzman behavior, was found). In the end ST2 itself is flawed, and Princeton admitted that the discussion of water was not significantly advanced through this drama. Is it worth it? Understanding the argument and its importance should be the focus.
  - YorkshireSeason 8y ago
    Sharing code is not always the answer Why let the perfect be the enemy of the good? Sharing code is not the full answer, especially in case of Monte-Carlo simulations in physics, because that kind of algorithm is hard to test: What is the testing oracle? What is the specification? But sharing code is part of the answer. Setting up a culture where it is unacceptable to submit a paper without open-sourcing the code and suitable testing (for simple edge ases), and suitable scripts that make reproducing the software simulations easy, is good scientific 'hygiene'. See for example [1, 2] for efforts towards reproducible software submissions in computer science. Reproduciblility is the very essence of the scientific method. [1] http://evaluate.inf.usi.ch/artifacts http://evaluate.inf.usi.ch/artifacts [2] http://www.artifact-eval.org/ http://www.artifact-eval.org/

ur-whale 8y ago

Modern science journals, in the same way they don't accept articles without proper reference to previous work, should not accept research articles unless there's a link to the source code in the article.

bigiain 8y ago

> “I had and was very willing to share the code,” he says. What he didn’t have, he says, was the time or personnel to prepare the code in a form that could be useful to an outsider. That's what this license is for: http://matt.might.net/articles/crapl/ http://matt.might.net/articles/crapl/

itronitron 8y ago
Considering that a paper is run through a subjective and highly critical peer-review I can see why authors would be reluctant to share their code.
- bigiain 8y ago
  It's an interesting observation then that the people doing the "highly critical peer review" might promote papers with no source code showing how the calculations were done over papers with messy-but-available source code. I know some fields intentionally/explicitly hide their working ("Law and sausage are two things you do not want to see being made."), but I wouldn't have thought academia would subscribe to that... There's a lot to be said for the IETF's policy of "rough consensus and running code".
  - itronitron 8y ago
    The paper should show how the calculations can be made, along with the results of that calculation, preferably on a publicly available dataset. A common strategy is to describe the calculation in highly convoluted or specialized language (mathematical formulas) in order to reduce the number of people that may try to reproduce the work. Having reviewed papers on multiple occasions, I wouldn't spend much if any time looking at the actual source code if it is available. Every paper submission is rushed because it needs to be ahead of what anyone else is doing (at least in CS and related fields).
    - bigiain 8y ago
      Do you see this as an inevitable "that's the way it is" part of how the peer review process works? Or as a flaw in the process? Or something in between. (A deeply cynical and suspicious part of me suspects this is because the prime motivator for "peer reviewed journals" is to make Elsiver richer, and academic knowledge sharing and reputation building is just a side effect of that...)