4 ms·

The problem with Mathematica from a science point of view is that, being closed source, means you can't independently ensure the calculations are happening corr

by gaius 9y ago

The problem with Mathematica from a science point of view is that, being closed source, means you can't independently ensure the calculations are happening correctly Have there been any high profile failures root-caused to Mathematica (or MATLAB or any similar product) getting its sums wrong? I can find any news stories etc. Plenty of serious calculations were and are done on “closed source” HP and TI calculators too. Every serious scientific instrument with its own data capture uses a binary blob too somewhere, so if that’s a problem for you then you can’t even trust the raw data! And even if you have all of the code - you still need to worry if the proprietary, closed FPU is working “correctly”. This sounds very much like a post-hoc justification for “its free as in beer”. Do you think Wikipedia is more trustworthy than real references too? What about blogs?

BlackFingolfin 9y ago
I am not sure that there are/were "high profile failures", but serious bugs for sure, not just in Mathmetica but also in the calculators you mention; ser eg here: https://mathoverflow.net/questions/11517/computer-algebra-errors https://mathoverflow.net/questions/11517/computer-algebra-er... Now being open source does not prevent such errors. What I think is far more important here is open issue tracking: companies like Wolfram may not be exactly eager to let you know that there are serious bugs in their products (and what they are) unless they must. Being able to fix the bug yourself can of course be a real perk for some user's, but realistically, it's out if question for most.
fdej 9y ago
Here is an article from a couple of years ago about how one group of mathematicians were misled by (and had to spend some time tracking down) a bug in Mathematica determinant evaluation: http://www.ams.org/notices/201410/rnoti-p1249.pdf http://www.ams.org/notices/201410/rnoti-p1249.pdf
- FractalLP 9y ago
  Thanks for posting. I'm sure bugs like this show up in any and all software. I do wish they had a public bug tracker though.
PopePompous 9y ago
Science publication is moving (very, very, very slowly) towards a model where instead of a final, polished traditional paper, the raw data along with the software tools and interpretation is published. In principle this should allow readers to completely understand and reproduce the processing of the raw data, rather than reading a few paragraphs summarizing the processing done by the authors. Using a closed source tool for processing the data limits how deeply a reader can delve into the processing that the authors did, because the functions in the proprietary package are black boxes. Jupyter has no black boxes.
- gaius 9y ago
  In principle this should allow readers to completely understand and reproduce the processing of the raw data, rather than reading a few paragraphs summarizing the processing done by the authors. Using a closed source tool But consider http://www.bbc.co.uk/news/science-environment-39054778 http://www.bbc.co.uk/news/science-environment-39054778 "Science is facing a "reproducibility crisis" where more than two-thirds of researchers have tried and failed to reproduce another scientist's experiments" I don't think that can be handwaved away as "OMG closed source software!". Especially since all the scientists in a given field will have access to the same software anyway. Give them open source and the issue will persist, and we both know it because the root cause isn't anything to do with the license of the software
  - nmca 9y ago
    Open source software, in general, tends to make things more reproducible. Sure, software licencing might not be the singular root cause, but why does that suggest we shouldn't capitalise on the improvements available?
    - gaius 9y ago
      Open source software, in general, tends to make things more reproducible Citation very much needed for that. Because you can very easily find that 6 months or a year later you update your dependencies and everything is now broken. I recently came back to a Python project after a year, updated my packages then realized: I simply cannot be bothered to unpick the mess that resulted just to add one trivial feature. Whereas the poster child for backwards compatibility is closed-source and proprietary. Science is not reproducible because there are no incentives for it to be so, despite everyone paying lip service to it. It's extra work and helps those who are competing with you for grants, after all. That's a social problem, not a software one. The software is irrelevant.
madengr 9y ago
I’ll add that many closed sourced developers will show you sections of the source code if you sign an NDA. I’ve done that on several occasions when I needed to see how a model was implemented. As long as you are not a competitor, it’s usually not an issue. I sign NDAs often to see IP.
- gaius 9y ago
  Additionally most of Mathematica is written in Mathematica and you can just read it. Same with MATLAB. I get the feeling that many of the Jupyter fans commenting here have only ever used a Jupyter setup and are unaware of the wider industry

by gaius 9y ago

BlackFingolfin 9y ago

I am not sure that there are/were "high profile failures", but serious bugs for sure, not just in Mathmetica but also in the calculators you mention; ser eg here: https://mathoverflow.net/questions/11517/computer-algebra-errors https://mathoverflow.net/questions/11517/computer-algebra-er... Now being open source does not prevent such errors. What I think is far more important here is open issue tracking: companies like Wolfram may not be exactly eager to let you know that there are serious bugs in their products (and what they are) unless they must. Being able to fix the bug yourself can of course be a real perk for some user's, but realistically, it's out if question for most.

fdej 9y ago

Here is an article from a couple of years ago about how one group of mathematicians were misled by (and had to spend some time tracking down) a bug in Mathematica determinant evaluation: http://www.ams.org/notices/201410/rnoti-p1249.pdf http://www.ams.org/notices/201410/rnoti-p1249.pdf

FractalLP 9y ago
Thanks for posting. I'm sure bugs like this show up in any and all software. I do wish they had a public bug tracker though.

PopePompous 9y ago

Science publication is moving (very, very, very slowly) towards a model where instead of a final, polished traditional paper, the raw data along with the software tools and interpretation is published. In principle this should allow readers to completely understand and reproduce the processing of the raw data, rather than reading a few paragraphs summarizing the processing done by the authors. Using a closed source tool for processing the data limits how deeply a reader can delve into the processing that the authors did, because the functions in the proprietary package are black boxes. Jupyter has no black boxes.

gaius 9y ago
In principle this should allow readers to completely understand and reproduce the processing of the raw data, rather than reading a few paragraphs summarizing the processing done by the authors. Using a closed source tool But consider http://www.bbc.co.uk/news/science-environment-39054778 http://www.bbc.co.uk/news/science-environment-39054778 "Science is facing a "reproducibility crisis" where more than two-thirds of researchers have tried and failed to reproduce another scientist's experiments" I don't think that can be handwaved away as "OMG closed source software!". Especially since all the scientists in a given field will have access to the same software anyway. Give them open source and the issue will persist, and we both know it because the root cause isn't anything to do with the license of the software
- nmca 9y ago
  Open source software, in general, tends to make things more reproducible. Sure, software licencing might not be the singular root cause, but why does that suggest we shouldn't capitalise on the improvements available?
  - gaius 9y ago
    Open source software, in general, tends to make things more reproducible Citation very much needed for that. Because you can very easily find that 6 months or a year later you update your dependencies and everything is now broken. I recently came back to a Python project after a year, updated my packages then realized: I simply cannot be bothered to unpick the mess that resulted just to add one trivial feature. Whereas the poster child for backwards compatibility is closed-source and proprietary. Science is not reproducible because there are no incentives for it to be so, despite everyone paying lip service to it. It's extra work and helps those who are competing with you for grants, after all. That's a social problem, not a software one. The software is irrelevant.

madengr 9y ago

I’ll add that many closed sourced developers will show you sections of the source code if you sign an NDA. I’ve done that on several occasions when I needed to see how a model was implemented. As long as you are not a competitor, it’s usually not an issue. I sign NDAs often to see IP.

gaius 9y ago
Additionally most of Mathematica is written in Mathematica and you can just read it. Same with MATLAB. I get the feeling that many of the Jupyter fans commenting here have only ever used a Jupyter setup and are unaware of the wider industry