4 ms·

Reproducible has a much simpler meaning than that you describe: it’s feasible for any scientist to recreate the result perpetually. While inspection of the meth

Reproducible has a much simpler meaning than that you describe: it’s feasible for any scientist to recreate the result perpetually. While inspection of the methods might help one understand the result—and I think itself holds a great deal of value to especially mathematicians—just the simple ability to compute the answers yourself without binding yourself to a contract or paying $1000’s of dollars in license fees is a valuable aspect to science. Especially so when the calculations are at the level of pure algebraic manipulation. Your (and Wolfram’s!) argument about “not needing to see the insides because really only 10 people in the world understand it anyway” should really be an argument for opening it up. If 10 people could understand it, and we must intersect that group with folks who can access the source, we are left in quite a dismal state of affairs.

vtail 7y ago
While I don't disagree with your definition of reproducibility, I want to point out that very few papers would satisfy that criteria, regardless of their use of a closed- vs open-source software. Here is a recent example: Imperial College COVID-19 response team published an article[1] where they modeled different effects of non-pharmaceutical interventions, such as suppression and mitigation on the number of infected, deaths, etc. This is a very interesting result, but it's impossible to replicate their results in practice without contacting the authors, as their methodology is not enough to reproduce it. While someone else[2] posted their own model that is very well documented and fully reproducible by anyone with a $230/year personal license. While theoretically [1] is high science and [2] is not, in my own opinion [2] is better than [1]. I would love more science to be done and discussed that way. Ideally, using open-source software, but in practice, using Wolfram Language, in that case, is already good enough in my opinion. PS. I'm not affiliated with Wolfram Research in any way. [1] https://www.imperial.ac.uk/media/imperial-college/medicine/sph/ide/gida-fellowships/Imperial-College-COVID19-NPI-modelling-16-03-2020.pdf?fbclid=IwAR3RwGVLlRhBGLT2_KdL1RXk6_XF04RFSYgz5MjhN6LLysKbGonXGnRiZOY https://www.imperial.ac.uk/media/imperial-college/medicine/s... [2] https://community.wolfram.com/groups/-/m/t/1901002 https://community.wolfram.com/groups/-/m/t/1901002
- lonelappde 7y ago
  Making an exact copy of an experiment is the lwest level of reproducibility. In science , reproducibility means reproducing the results with different components (people, tools, methodology), showing robust Independence form potential confounders.
vtail 7y ago
Regarding your second point - I don't have a horse in that race, other than tremendously enjoying my personal license that I pay for out of my pocket; but I found their arguments[1] quite reasonable. I think there was an HN discussion about this some time ago. [1] https://blog.wolfram.com/2019/04/02/why-wolfram-tech-isnt-open-source-a-dozen-reasons/ https://blog.wolfram.com/2019/04/02/why-wolfram-tech-isnt-op...
- reikonomusha 7y ago
  I find the arguments confused and lacking. Many of those arguments conflate “open source” with “community driven”. Rarely did they make a compelling case as to how availability of the source code would hurt the customers. And many, many amazing open source teams have demonstrated that even including community driven design in some cases does not hamper the quality of the product. Look at Apple and Swift, or Rust, as examples. I read their arguments as “we do not know how to engage with a community effort because it’s not in our company DNA” under the facade of supposedly legitimate reasons.
  - vtail 7y ago
    Swift is a great example, but don't forget that is essentially funded by Apple. It would look very different if it would be a true community effort. Rust I'm not very familiar with: how much of their funding is dependent on Mozilla, at least for the core developer(s)? Also, a language + a standard library has a (much!) smaller surface comparing to Wolfram's vision.
    - reikonomusha 7y ago
      They have funding; that’s great! Exactly what I’m saying is that these projects manage to be community run (to some extent) while still remaining successful. If Wolfram’s reason to not publish the source code is that they believe they would no longer have a business model, they should list that as the One Reason, not 12 gaslighting reasons.
      - 7y ago

vtail 7y ago

While I don't disagree with your definition of reproducibility, I want to point out that very few papers would satisfy that criteria, regardless of their use of a closed- vs open-source software. Here is a recent example: Imperial College COVID-19 response team published an article[1] where they modeled different effects of non-pharmaceutical interventions, such as suppression and mitigation on the number of infected, deaths, etc. This is a very interesting result, but it's impossible to replicate their results in practice without contacting the authors, as their methodology is not enough to reproduce it. While someone else[2] posted their own model that is very well documented and fully reproducible by anyone with a $230/year personal license. While theoretically [1] is high science and [2] is not, in my own opinion [2] is better than [1]. I would love more science to be done and discussed that way. Ideally, using open-source software, but in practice, using Wolfram Language, in that case, is already good enough in my opinion. PS. I'm not affiliated with Wolfram Research in any way. [1] https://www.imperial.ac.uk/media/imperial-college/medicine/sph/ide/gida-fellowships/Imperial-College-COVID19-NPI-modelling-16-03-2020.pdf?fbclid=IwAR3RwGVLlRhBGLT2_KdL1RXk6_XF04RFSYgz5MjhN6LLysKbGonXGnRiZOY https://www.imperial.ac.uk/media/imperial-college/medicine/s... [2] https://community.wolfram.com/groups/-/m/t/1901002 https://community.wolfram.com/groups/-/m/t/1901002

lonelappde 7y ago
Making an exact copy of an experiment is the lwest level of reproducibility. In science , reproducibility means reproducing the results with different components (people, tools, methodology), showing robust Independence form potential confounders.

vtail 7y ago

Regarding your second point - I don't have a horse in that race, other than tremendously enjoying my personal license that I pay for out of my pocket; but I found their arguments[1] quite reasonable. I think there was an HN discussion about this some time ago. [1] https://blog.wolfram.com/2019/04/02/why-wolfram-tech-isnt-open-source-a-dozen-reasons/ https://blog.wolfram.com/2019/04/02/why-wolfram-tech-isnt-op...

reikonomusha 7y ago
I find the arguments confused and lacking. Many of those arguments conflate “open source” with “community driven”. Rarely did they make a compelling case as to how availability of the source code would hurt the customers. And many, many amazing open source teams have demonstrated that even including community driven design in some cases does not hamper the quality of the product. Look at Apple and Swift, or Rust, as examples. I read their arguments as “we do not know how to engage with a community effort because it’s not in our company DNA” under the facade of supposedly legitimate reasons.
- vtail 7y ago
  Swift is a great example, but don't forget that is essentially funded by Apple. It would look very different if it would be a true community effort. Rust I'm not very familiar with: how much of their funding is dependent on Mozilla, at least for the core developer(s)? Also, a language + a standard library has a (much!) smaller surface comparing to Wolfram's vision.
  - reikonomusha 7y ago
    They have funding; that’s great! Exactly what I’m saying is that these projects manage to be community run (to some extent) while still remaining successful. If Wolfram’s reason to not publish the source code is that they believe they would no longer have a business model, they should list that as the One Reason, not 12 gaslighting reasons.
    - 7y ago