5 ms·

I am not surprised that people find errors in code written by researchers and grad students who have little training in software development and, perhaps more i

by lutorm 17y ago

I am not surprised that people find errors in code written by researchers and grad students who have little training in software development and, perhaps more importantly, are doing so in a culture which values them writing papers, not good code. (See for example http://lanl.arxiv.org/abs/0903.3971 http://lanl.arxiv.org/abs/0903.3971 for a discussion of this situation in astronomy/astrophysics.) I find it much more surprising that professionally developed software used for scientific research is also error ridden. And while it might be difficult to convince individual researchers to release their code, that's nothing compared to the difficulties of convincing Wolfram research to release the source code to Mathematica... But I do think that research is somewhat undeservedly singled out for this, just because some academic software is open for inspection. Like the article mentions, it certainly seems like the financial software has caused a lot of badness. How about flight control software used by NASA that crashed the Mars orbiter? Who knows how many innocent lives have been lost due to software errors in military systems like UAVs and missiles. Maybe none, but we can't know because it's all secret. Shouldn't they be required to show their code, too?

smallblacksun 17y ago
But the military and NASA don't claim to be generating reproducible knowledge through the use of their code. In particular, the military doesn't WANT other people to be able to reproduce what their code does. Also, there is a difference between operational code (code that runs a physical object like a lander or a UAV) and analytical code. NASA makes some of their code available here: http://opensource.arc.nasa.gov http://opensource.arc.nasa.gov
- lutorm 17y ago
  Cool, I didn't know about the NASA open source project. You are right that knowledge production isn't the purpose of those other entities, of course. However, in my mind the purpose is less important than the outcome -- why is it more harmful to society if scientists produce a flawed scientific result than if the military kills innocents or the financial sector brings on a market crash because of flawed models? They all hurt society and could all benefit from more scrutiny. I admit the military case is a stretch, but certainly the financial sector seems like a relevant example.
DaniFong 17y ago
Mathematica probably has 3 orders of magnitude more code coverage than code underlying scientific papers. What's more, WRI encourages people to publish interactive Mathematica notebooks as a kind of literate programming that allows people to run code as well as read and view the results. I think it's an overall net positive.
- cgranade 17y ago
  That is an overall net positive, but I still think that, in order to be fully compatible with the ideals of academic freedom and reproducibility, papers should depend largely (ideally, only) on code that is fully open source. I recognize that this may not be compatible with the interests of companies like Wolfram and Math Works, but I think it is in the best long-term interests of research as a whole.
  - DaniFong 17y ago
    I think so too. Patrick Collison and I and likely many others have ideas for an open source Mathematica competitor in idea stages and design, and in the back of our minds. But it's a herculean task. Maybe one day we will get to it.
    - cgranade 17y ago
      It is a Herculean task, but that's part of the miracle and promise of open-source: you don't have to do it alone. To address the lack of a good MATLAB-style science oriented IDE, I started a project (http://github.com/cgranade/scicore http://github.com/cgranade/scicore) today to try and attract more intelligent coders than myself to help. As others have pointed out, SAGE (http://www.sagemath.org/ http://www.sagemath.org/) is a good effort at an open-source Mathematica competitor, even if I feel it lacks something in the UI department. Point being, it's possible to get a project going that becomes larger than yourself using an open model.

by lutorm 17y ago

smallblacksun 17y ago

But the military and NASA don't claim to be generating reproducible knowledge through the use of their code. In particular, the military doesn't WANT other people to be able to reproduce what their code does. Also, there is a difference between operational code (code that runs a physical object like a lander or a UAV) and analytical code. NASA makes some of their code available here: http://opensource.arc.nasa.gov http://opensource.arc.nasa.gov

lutorm 17y ago
Cool, I didn't know about the NASA open source project. You are right that knowledge production isn't the purpose of those other entities, of course. However, in my mind the purpose is less important than the outcome -- why is it more harmful to society if scientists produce a flawed scientific result than if the military kills innocents or the financial sector brings on a market crash because of flawed models? They all hurt society and could all benefit from more scrutiny. I admit the military case is a stretch, but certainly the financial sector seems like a relevant example.

DaniFong 17y ago

Mathematica probably has 3 orders of magnitude more code coverage than code underlying scientific papers. What's more, WRI encourages people to publish interactive Mathematica notebooks as a kind of literate programming that allows people to run code as well as read and view the results. I think it's an overall net positive.

cgranade 17y ago
That is an overall net positive, but I still think that, in order to be fully compatible with the ideals of academic freedom and reproducibility, papers should depend largely (ideally, only) on code that is fully open source. I recognize that this may not be compatible with the interests of companies like Wolfram and Math Works, but I think it is in the best long-term interests of research as a whole.
- DaniFong 17y ago
  I think so too. Patrick Collison and I and likely many others have ideas for an open source Mathematica competitor in idea stages and design, and in the back of our minds. But it's a herculean task. Maybe one day we will get to it.
  - cgranade 17y ago
    It is a Herculean task, but that's part of the miracle and promise of open-source: you don't have to do it alone. To address the lack of a good MATLAB-style science oriented IDE, I started a project (http://github.com/cgranade/scicore http://github.com/cgranade/scicore) today to try and attract more intelligent coders than myself to help. As others have pointed out, SAGE (http://www.sagemath.org/ http://www.sagemath.org/) is a good effort at an open-source Mathematica competitor, even if I feel it lacks something in the UI department. Point being, it's possible to get a project going that becomes larger than yourself using an open model.