5 ms·

> You don't need source to audit the results. In fact, you do. Modern computer mathematical results require full exposure, nothing can be left to the imaginati

by lutusp 13y ago

> You don't need source to audit the results. In fact, you do. Modern computer mathematical results require full exposure, nothing can be left to the imagination. This is true and understandable. Going back to the first important computer mathematical result (the solution to the four-color map conjecture¹ in 1976), to be taken seriously, computer math results have been required to be fully transparent. Imagine auditing a cancer cure without knowing anything about the methods or tools. Same idea. 1. http://en.wikipedia.org/wiki/Four_color_theorem http://en.wikipedia.org/wiki/Four_color_theorem

taliesinb 13y ago
I actually somewhat sympathize with your view. But I think you overstate the problem: if someone discovers something with Mathematica, they can then formalize it with an open proof assistant like Coq before they publish, and modern journals in fact encourage this sort of thing. Really, it's not a zero-sum-game between open-source and commercial tools.
- jordigh 13y ago
  "Open source" and "commercial" are not antonyms, but people working on proprietary software think they are.
  - Someone 13y ago
    Some people may think that, but I doubt that that applies here. Wolfram certainly knows that "proprietary" and "free" aren't antonyms. I do think there is a commercial side to this, too, however. This is a kind of a hook to catch future customers of the commercial package.
  - comex 13y ago
    Only if you use an overly literal interpretation of "commercial" yet don't use the literal interpretation of "open source" (since it's also possible to make source code visible to the public without using a license that would be considered "open source").
    - jlgreco 13y ago
      You have just rediscovered the reason that some people talk about "free software", and some people talk about "open source". The sort of access to source code that is being expected here is really no more than the sort of access that you can get to Windows code. It really is not an unreasonable expectation.
      - comex 13y ago
        No - when most people use the term "open source", they mean something along the lines of the Open Source Definition (http://opensource.org/definition http://opensource.org/definition), rather than the literal meaning; this is independent from the distinctions between open source and free software, which are mostly about philosophy rather than the set of applicable licenses (http://www.gnu.org/philosophy/free-software-for-freedom.html http://www.gnu.org/philosophy/free-software-for-freedom.html) I suppose that you, however, do not, which makes what I was trying to say irrelevant.
  - throwawaykf 13y ago
    And a lot of people using open source products think so too :-)
- lutusp 13y ago
  > Really, it's not a zero-sum-game between open-source and commercial tools. Yes, there are mathematical results that can be validated independently. But the problem is not so much pure mathematics as numerical results. In that arena, the method by which the results were acquired must be accessible in detail. And much of modern applied mathematics relies on numerical results. Imagine producing a result of some value and being unable to explain in detail how it was arrived at, because it emanated from a closed-source program. The four-color map problem I mentioned earlier was the first important case where this problem had to be dealt with, and it was resolved to everyone's satisfaction only because the full source was made available.
  - taliesinb 13y ago
    You're jumping around a bit.. Did the four-color map problem use numerical results? I imagine it was just a pure combinatorial enumeration problem (I have not read the original paper, I may be wrong about that). If you're doing a finite element method to validate a bridge design, you're hardly publishing a paper based on the result (though of course you will do plenty of validation and then times everything by 2 for good measure!). What kinds of actual fields of mathematics are you thinking about, here? I know that error analysis is a deep and arcane subject, and personally I have no idea how well or badly we do in that department, but I'm struggling to think of where the details of how we get it wrong or right is crucial for the development of published academic mathematics. "Most" maths is totally abstract these days, categories and sheaves and topi and what not.
    - lutusp 13y ago
      > You're jumping around a bit. In that case, anyone who uses more than one example to make his point is jumping around. If allowed to prevail, this non-argument would cripple debate. > Did the four-color map problem use numerical results? Yes, emphatically so. There were a certain finite number of possible maps (1936 to be precise), and each of them had to be tested explicitly. The reason for the requirement for full disclosure was that the tested set needed to be examined to assure that nothing germane was excluded, nothing included on specious grounds, and the manner of testing had to be closely examined. None of this could be reduced to a simple equation, and none of it yielded to a formal, concise mathematical proof. > I imagine it was just a pure combinatorial enumeration problem (I have not read the original paper, I may be wrong about that). If that were true, there would have been no role for a computer program -- it would have been published as a pure and formal result, like Fermat's last theorem. No one would have cared about the role played by computers, because a formal and pure proof could be examined by anyone sufficiently versed in mathematics. > What kinds of actual fields of mathematics are you thinking about, here? Any field that would benefit from use of a computer. By recent count, that's nearly all mathematical fields for one reason or another, including areas we once might have thought would have no use for a computer. Computer mathematics has gotten to the point where computers, laboring independently, come up with new theorems that humans have overlooked. > I know that error analysis is a deep and arcane subject, and personally I have no idea how well or badly we do in that department, but I'm struggling to think of where the details of how we get it wrong or right is crucial for the development of published academic mathematics. Most struggle with that, and part of the struggle is being able to see the detail of the algorithms used to arrive at a result. No detail, no trust, no science. > "Most" maths is totally abstract these days ... Not relevant. Relevant would be whether that abstract mathematics is or is not fully revealed in its details. For science, it must be. This article -- http://www.newscientist.com/article/dn7286-computer-generates-verifiable-mathematics-proof.html#.Uo8YZkpDsrI http://www.newscientist.com/article/dn7286-computer-generate... -- discusses the role of theorem-proving software like Coq, but includes a misleading statement by someone at Microsoft: "Microsoft hopes to develop a similar system for checking the logic used in computer programs, which could pre-empt some unforeseen bugs that cause programs to crash." I hope this is an error on the part of the journalist who wrote the article, because it implies that Turing's halting problem, and Gödel's incompleteness theorems, which lie behind it, are potentially soluble. They aren't. All the more reason for full disclosure.
thinkpad20 13y ago
If what one is publishing is an algorithm or formula or what-have-you, it's only required that the algorithm itself be fully exposed. It particular implementation, much less so the implementation of the particular software that implementation is written in, is only incidental. When you're giving a driving test, you don't need to know how a fuel-injection system works.
- lutusp 13y ago
  > It particular implementation, much less so the implementation of the particular software that implementation is written in, is only incidental. Not at all. In computer programming, examples abound in which the gap between the formal algorithm and its practical embodiment led to catastrophe. One can scarcely count the number of times in which the specific manner by which a simply stated algorithm was implemented needed to be examined more closely: http://en.wikipedia.org/wiki/Ariane_5 http://en.wikipedia.org/wiki/Ariane_5 "Ariane 5's first test flight (Ariane 5 Flight 501) on 4 June 1996 failed, with the rocket self-destructing 37 seconds after launch because of a malfunction in the control software.[20] A data conversion from 64-bit floating point value to 16-bit signed integer value to be stored in a variable representing horizontal bias caused a processor trap (operand error) ..." > When you're giving a driving test, you don't need to know how a fuel-injection system works. No, but someone does. If the software is inaccessible, then no one can check it except its owners, who have an incentive not to check it thoroughly, or to conceal what they have discovered.
blhack 13y ago
Out of curiosity: how deep does that need go? Low level hardware analysis? Chip manufacturing facility audits? How would you even approach this?
- noonespecial 13y ago
  Recalling the The Pentium FDIV bug, I'd say yes. Chip level.
- lutusp 13y ago
  > Out of curiosity: how deep does that need go? Deep enough that the meaning of the result cannot reasonably be called into question, just like in science. > How would you even approach this? As scientists. Someone once described science as ... not so much knowing as knowing that we know. Same idea. Many results that seemed certain when published, evaporated once the mathematics were examined closely. This happens even when the entire process -- data, equations, conclusions -- are all included in the publication. Given that history, imagine how much more cautious people want to be when part of the process of arriving at a critical conclusion is concealed perpetually.
  - mkingston 13y ago
    "Deep enough that the meaning of the result cannot reasonably be called into question, just like in science." I think what is really being asked is: "What level of confidence is required?". I would assume you trust the hardware and only scrutinise the software? In contrast: are results tested on a plurality of machines with intentionally, measurably different hardware and the absence of discrepancies implies that the hardware is not a factor with regard to the problem being solved? Presumedly you'd need some sort of standardised test suite to pass on your hardware to really confidently declare the hardware to not be a factor. I doubt this is the case, but like the previous poster, I'm quite curious as to the specific level of scrutiny applied here.
    - lutusp 13y ago
      > I think what is really being asked is: "What level of confidence is required?". That's something that varies from field to field. In physics, a new discovery requires a five-sigma result, meaning a result that has a one in 3.5 million probability of arising by chance rather than the legitimate detection of a new particle or phenomenon. http://www.select-statistics.co.uk/blog/webmaster/one-sigma-or-five http://www.select-statistics.co.uk/blog/webmaster/one-sigma-... In psychology, by contrast, a p-value of 0.05 (i.e. 2 sigma) is accepted as statistically significant, which by itself may explain the sad state of the field. http://en.wikipedia.org/wiki/P-value http://en.wikipedia.org/wiki/P-value > I would assume you trust the hardware and only scrutinise the software? No, a scientist doesn't trust either hardware or software, but hardware can be checked and validated. In the case of Mathematica, the software cannot be checked and validated, it must be trusted. And for a scientist, trust cannot be distinguished from faith.

by lutusp 13y ago

taliesinb 13y ago

I actually somewhat sympathize with your view. But I think you overstate the problem: if someone discovers something with Mathematica, they can then formalize it with an open proof assistant like Coq before they publish, and modern journals in fact encourage this sort of thing. Really, it's not a zero-sum-game between open-source and commercial tools.

jordigh 13y ago
"Open source" and "commercial" are not antonyms, but people working on proprietary software think they are.
- Someone 13y ago
  Some people may think that, but I doubt that that applies here. Wolfram certainly knows that "proprietary" and "free" aren't antonyms. I do think there is a commercial side to this, too, however. This is a kind of a hook to catch future customers of the commercial package.
- comex 13y ago
  Only if you use an overly literal interpretation of "commercial" yet don't use the literal interpretation of "open source" (since it's also possible to make source code visible to the public without using a license that would be considered "open source").
  - jlgreco 13y ago
    You have just rediscovered the reason that some people talk about "free software", and some people talk about "open source". The sort of access to source code that is being expected here is really no more than the sort of access that you can get to Windows code. It really is not an unreasonable expectation.
    - comex 13y ago
      No - when most people use the term "open source", they mean something along the lines of the Open Source Definition (http://opensource.org/definition http://opensource.org/definition), rather than the literal meaning; this is independent from the distinctions between open source and free software, which are mostly about philosophy rather than the set of applicable licenses (http://www.gnu.org/philosophy/free-software-for-freedom.html http://www.gnu.org/philosophy/free-software-for-freedom.html) I suppose that you, however, do not, which makes what I was trying to say irrelevant.
- throwawaykf 13y ago
  And a lot of people using open source products think so too :-)
lutusp 13y ago
> Really, it's not a zero-sum-game between open-source and commercial tools. Yes, there are mathematical results that can be validated independently. But the problem is not so much pure mathematics as numerical results. In that arena, the method by which the results were acquired must be accessible in detail. And much of modern applied mathematics relies on numerical results. Imagine producing a result of some value and being unable to explain in detail how it was arrived at, because it emanated from a closed-source program. The four-color map problem I mentioned earlier was the first important case where this problem had to be dealt with, and it was resolved to everyone's satisfaction only because the full source was made available.
- taliesinb 13y ago
  You're jumping around a bit.. Did the four-color map problem use numerical results? I imagine it was just a pure combinatorial enumeration problem (I have not read the original paper, I may be wrong about that). If you're doing a finite element method to validate a bridge design, you're hardly publishing a paper based on the result (though of course you will do plenty of validation and then times everything by 2 for good measure!). What kinds of actual fields of mathematics are you thinking about, here? I know that error analysis is a deep and arcane subject, and personally I have no idea how well or badly we do in that department, but I'm struggling to think of where the details of how we get it wrong or right is crucial for the development of published academic mathematics. "Most" maths is totally abstract these days, categories and sheaves and topi and what not.
  - lutusp 13y ago
    > You're jumping around a bit. In that case, anyone who uses more than one example to make his point is jumping around. If allowed to prevail, this non-argument would cripple debate. > Did the four-color map problem use numerical results? Yes, emphatically so. There were a certain finite number of possible maps (1936 to be precise), and each of them had to be tested explicitly. The reason for the requirement for full disclosure was that the tested set needed to be examined to assure that nothing germane was excluded, nothing included on specious grounds, and the manner of testing had to be closely examined. None of this could be reduced to a simple equation, and none of it yielded to a formal, concise mathematical proof. > I imagine it was just a pure combinatorial enumeration problem (I have not read the original paper, I may be wrong about that). If that were true, there would have been no role for a computer program -- it would have been published as a pure and formal result, like Fermat's last theorem. No one would have cared about the role played by computers, because a formal and pure proof could be examined by anyone sufficiently versed in mathematics. > What kinds of actual fields of mathematics are you thinking about, here? Any field that would benefit from use of a computer. By recent count, that's nearly all mathematical fields for one reason or another, including areas we once might have thought would have no use for a computer. Computer mathematics has gotten to the point where computers, laboring independently, come up with new theorems that humans have overlooked. > I know that error analysis is a deep and arcane subject, and personally I have no idea how well or badly we do in that department, but I'm struggling to think of where the details of how we get it wrong or right is crucial for the development of published academic mathematics. Most struggle with that, and part of the struggle is being able to see the detail of the algorithms used to arrive at a result. No detail, no trust, no science. > "Most" maths is totally abstract these days ... Not relevant. Relevant would be whether that abstract mathematics is or is not fully revealed in its details. For science, it must be. This article -- http://www.newscientist.com/article/dn7286-computer-generates-verifiable-mathematics-proof.html#.Uo8YZkpDsrI http://www.newscientist.com/article/dn7286-computer-generate... -- discusses the role of theorem-proving software like Coq, but includes a misleading statement by someone at Microsoft: "Microsoft hopes to develop a similar system for checking the logic used in computer programs, which could pre-empt some unforeseen bugs that cause programs to crash." I hope this is an error on the part of the journalist who wrote the article, because it implies that Turing's halting problem, and Gödel's incompleteness theorems, which lie behind it, are potentially soluble. They aren't. All the more reason for full disclosure.

thinkpad20 13y ago

If what one is publishing is an algorithm or formula or what-have-you, it's only required that the algorithm itself be fully exposed. It particular implementation, much less so the implementation of the particular software that implementation is written in, is only incidental. When you're giving a driving test, you don't need to know how a fuel-injection system works.

lutusp 13y ago
> It particular implementation, much less so the implementation of the particular software that implementation is written in, is only incidental. Not at all. In computer programming, examples abound in which the gap between the formal algorithm and its practical embodiment led to catastrophe. One can scarcely count the number of times in which the specific manner by which a simply stated algorithm was implemented needed to be examined more closely: http://en.wikipedia.org/wiki/Ariane_5 http://en.wikipedia.org/wiki/Ariane_5 "Ariane 5's first test flight (Ariane 5 Flight 501) on 4 June 1996 failed, with the rocket self-destructing 37 seconds after launch because of a malfunction in the control software.[20] A data conversion from 64-bit floating point value to 16-bit signed integer value to be stored in a variable representing horizontal bias caused a processor trap (operand error) ..." > When you're giving a driving test, you don't need to know how a fuel-injection system works. No, but someone does. If the software is inaccessible, then no one can check it except its owners, who have an incentive not to check it thoroughly, or to conceal what they have discovered.

blhack 13y ago

Out of curiosity: how deep does that need go? Low level hardware analysis? Chip manufacturing facility audits? How would you even approach this?

noonespecial 13y ago
Recalling the The Pentium FDIV bug, I'd say yes. Chip level.
lutusp 13y ago
> Out of curiosity: how deep does that need go? Deep enough that the meaning of the result cannot reasonably be called into question, just like in science. > How would you even approach this? As scientists. Someone once described science as ... not so much knowing as knowing that we know. Same idea. Many results that seemed certain when published, evaporated once the mathematics were examined closely. This happens even when the entire process -- data, equations, conclusions -- are all included in the publication. Given that history, imagine how much more cautious people want to be when part of the process of arriving at a critical conclusion is concealed perpetually.
- mkingston 13y ago
  "Deep enough that the meaning of the result cannot reasonably be called into question, just like in science." I think what is really being asked is: "What level of confidence is required?". I would assume you trust the hardware and only scrutinise the software? In contrast: are results tested on a plurality of machines with intentionally, measurably different hardware and the absence of discrepancies implies that the hardware is not a factor with regard to the problem being solved? Presumedly you'd need some sort of standardised test suite to pass on your hardware to really confidently declare the hardware to not be a factor. I doubt this is the case, but like the previous poster, I'm quite curious as to the specific level of scrutiny applied here.
  - lutusp 13y ago
    > I think what is really being asked is: "What level of confidence is required?". That's something that varies from field to field. In physics, a new discovery requires a five-sigma result, meaning a result that has a one in 3.5 million probability of arising by chance rather than the legitimate detection of a new particle or phenomenon. http://www.select-statistics.co.uk/blog/webmaster/one-sigma-or-five http://www.select-statistics.co.uk/blog/webmaster/one-sigma-... In psychology, by contrast, a p-value of 0.05 (i.e. 2 sigma) is accepted as statistically significant, which by itself may explain the sad state of the field. http://en.wikipedia.org/wiki/P-value http://en.wikipedia.org/wiki/P-value > I would assume you trust the hardware and only scrutinise the software? No, a scientist doesn't trust either hardware or software, but hardware can be checked and validated. In the case of Mathematica, the software cannot be checked and validated, it must be trusted. And for a scientist, trust cannot be distinguished from faith.