3 ms·

Many people on this and similar threads rightly question whether results received with the help of a closed source software are reproducible. I think it's a fai

by vtail 7y ago

Many people on this and similar threads rightly question whether results received with the help of a closed source software are reproducible. I think it's a fair question, but these discussions often miss a couple of important points. First, there is a question of theoretical vs. practical reproducibility: true, in _theory_, an open-source codebase with many millions of lines of code of highly complex transformation can be checked for accuracy by anyone. In _practice_, only very few insiders will have both technical understanding and time to verify the correctness of this or that algorithm, and everyone else would need to rely on their expertise. That situation is no different from using a proprietary system if we trust their authors. Second, while it might be shocking for someone, most of the science, outside of maybe math, it's not _practically_ reproducible. Most of the articles are behind paywalls, don't have enough data to replicate their results (even economists much more often than not won't include their raw data or algorithms or both). Also, no one who is trying to build a scientific career would try to replicate someone else's results, especially if it requires some costly equipment, reagents, etc. - the rewards in the scientific community are for novel results. Replication crisis[1] describes this pretty well. Most of the software that runs LHC at CERN is probably not inspectable by a regular Joe the physicist, yet somehow the physicist community somehow trusts their results, etc. In summary, while an open-source software of Mathematica quality would be awesome in theory, I highly doubt it would be practical for a long time. [1] https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis

reikonomusha 7y ago
Reproducible has a much simpler meaning than that you describe: it’s feasible for any scientist to recreate the result perpetually. While inspection of the methods might help one understand the result—and I think itself holds a great deal of value to especially mathematicians—just the simple ability to compute the answers yourself without binding yourself to a contract or paying $1000’s of dollars in license fees is a valuable aspect to science. Especially so when the calculations are at the level of pure algebraic manipulation. Your (and Wolfram’s!) argument about “not needing to see the insides because really only 10 people in the world understand it anyway” should really be an argument for opening it up. If 10 people could understand it, and we must intersect that group with folks who can access the source, we are left in quite a dismal state of affairs.
- vtail 7y ago
  While I don't disagree with your definition of reproducibility, I want to point out that very few papers would satisfy that criteria, regardless of their use of a closed- vs open-source software. Here is a recent example: Imperial College COVID-19 response team published an article[1] where they modeled different effects of non-pharmaceutical interventions, such as suppression and mitigation on the number of infected, deaths, etc. This is a very interesting result, but it's impossible to replicate their results in practice without contacting the authors, as their methodology is not enough to reproduce it. While someone else[2] posted their own model that is very well documented and fully reproducible by anyone with a $230/year personal license. While theoretically [1] is high science and [2] is not, in my own opinion [2] is better than [1]. I would love more science to be done and discussed that way. Ideally, using open-source software, but in practice, using Wolfram Language, in that case, is already good enough in my opinion. PS. I'm not affiliated with Wolfram Research in any way. [1] https://www.imperial.ac.uk/media/imperial-college/medicine/sph/ide/gida-fellowships/Imperial-College-COVID19-NPI-modelling-16-03-2020.pdf?fbclid=IwAR3RwGVLlRhBGLT2_KdL1RXk6_XF04RFSYgz5MjhN6LLysKbGonXGnRiZOY https://www.imperial.ac.uk/media/imperial-college/medicine/s... [2] https://community.wolfram.com/groups/-/m/t/1901002 https://community.wolfram.com/groups/-/m/t/1901002
  - lonelappde 7y ago
    Making an exact copy of an experiment is the lwest level of reproducibility. In science , reproducibility means reproducing the results with different components (people, tools, methodology), showing robust Independence form potential confounders.
- vtail 7y ago
  Regarding your second point - I don't have a horse in that race, other than tremendously enjoying my personal license that I pay for out of my pocket; but I found their arguments[1] quite reasonable. I think there was an HN discussion about this some time ago. [1] https://blog.wolfram.com/2019/04/02/why-wolfram-tech-isnt-open-source-a-dozen-reasons/ https://blog.wolfram.com/2019/04/02/why-wolfram-tech-isnt-op...

by vtail 7y ago

reikonomusha 7y ago

Reproducible has a much simpler meaning than that you describe: it’s feasible for any scientist to recreate the result perpetually. While inspection of the methods might help one understand the result—and I think itself holds a great deal of value to especially mathematicians—just the simple ability to compute the answers yourself without binding yourself to a contract or paying $1000’s of dollars in license fees is a valuable aspect to science. Especially so when the calculations are at the level of pure algebraic manipulation. Your (and Wolfram’s!) argument about “not needing to see the insides because really only 10 people in the world understand it anyway” should really be an argument for opening it up. If 10 people could understand it, and we must intersect that group with folks who can access the source, we are left in quite a dismal state of affairs.

vtail 7y ago
While I don't disagree with your definition of reproducibility, I want to point out that very few papers would satisfy that criteria, regardless of their use of a closed- vs open-source software. Here is a recent example: Imperial College COVID-19 response team published an article[1] where they modeled different effects of non-pharmaceutical interventions, such as suppression and mitigation on the number of infected, deaths, etc. This is a very interesting result, but it's impossible to replicate their results in practice without contacting the authors, as their methodology is not enough to reproduce it. While someone else[2] posted their own model that is very well documented and fully reproducible by anyone with a $230/year personal license. While theoretically [1] is high science and [2] is not, in my own opinion [2] is better than [1]. I would love more science to be done and discussed that way. Ideally, using open-source software, but in practice, using Wolfram Language, in that case, is already good enough in my opinion. PS. I'm not affiliated with Wolfram Research in any way. [1] https://www.imperial.ac.uk/media/imperial-college/medicine/sph/ide/gida-fellowships/Imperial-College-COVID19-NPI-modelling-16-03-2020.pdf?fbclid=IwAR3RwGVLlRhBGLT2_KdL1RXk6_XF04RFSYgz5MjhN6LLysKbGonXGnRiZOY https://www.imperial.ac.uk/media/imperial-college/medicine/s... [2] https://community.wolfram.com/groups/-/m/t/1901002 https://community.wolfram.com/groups/-/m/t/1901002
- lonelappde 7y ago
  Making an exact copy of an experiment is the lwest level of reproducibility. In science , reproducibility means reproducing the results with different components (people, tools, methodology), showing robust Independence form potential confounders.
vtail 7y ago
Regarding your second point - I don't have a horse in that race, other than tremendously enjoying my personal license that I pay for out of my pocket; but I found their arguments[1] quite reasonable. I think there was an HN discussion about this some time ago. [1] https://blog.wolfram.com/2019/04/02/why-wolfram-tech-isnt-open-source-a-dozen-reasons/ https://blog.wolfram.com/2019/04/02/why-wolfram-tech-isnt-op...