4 ms·

Here is where you're wrong: > You can't have a replication crisis inside of a deductive framework without changing the framework. You may change the framework

Here is where you're wrong: > You can't have a replication crisis inside of a deductive framework without changing the framework. You may change the framework, but the replication crisis would be that people don't notice it's changed. A proof can get so huge and complex that it would take a lifetime of study to understand it. One of the article's good points was that "you can replicate a math paper by reading it", but if you cannot read a proof, you cannot replicate it. If nobody can understand it, nobody can replicate it. There would be a big crisis if people started trusting huge papers blindly and using their results without stating them as assumptions. Mathematicians are largely not doing that, so there isn't really a crisis. But they might! And you would not notice that the framework had become inconsistent for a while. Therein would lie the crisis. It is nice to think that as soon as the framework changed, people would notice, because surely if it the ground moved, that would be obvious! But it is not true. Nothing about the deductive-ness or in-framework-ness of mathematics changes this. It is the same for psychology, the replication crisis is a crisis because people do not notice these results are wrong, and then rely on them in clinical guidelines and affect people's lives for generations. It is possible to stave off replication crises by using a machine, because mathematics is self-contained and you can use a computer to show that you proved something using only these specific assumptions and they can believe you without reading and understanding it fully. But this doesn't mean mathematics cannot ever have a replication crisis. I think your arguments here are best summarised as confusing "replication crisis" with "any published results being inconsistent with previous results". Your idea of "inconsistent" is also strange, the article considers this to mean mistakes in proofs, whereas you mean new mathematics that changes the field by discovery. You're just talking about something completely foreign from the concept of a replication crisis.

inglor_cz 5y ago
This scenario was actually pretty close when Andrew Wiles published his proof of the Last Fermat Theorem. The proof was huge, IIRC over 100 pages of maths, some of it pretty obscure, and verifying it took a lot of work of several people. They even found a flaw in it, but Wiles was able to fix it after a year.
- benibela 5y ago
  Or now the "proof" of the abc conjecture with the inter-universal Teichmüller theory
scoofy 5y ago
I’m talking about an event like the incompleteness theorem that ends aspirations like those of Russell and Whitehead. Paradoxes that lead to new axioms like in the development of set theory. Deductive frameworks can’t be wrong in the way that experimental results can be statistical anomalies. The author conflates errors with perfectly reasonable experimental results later shown to be anomalous upon replication.
- cormacrelf 5y ago
  Again you are talking about the framework as if it is exactly the same thing as the papers that attempt to describe it. It is not. Your statement that "deductive frameworks can't be wrong" is irrelevant to the question of whether a math paper can be wrong. If I prove something, it can be wrong, and it may turn out that I haven't proved it at all, but I can still call it a proof, convince other people it's correct, get it published in a journal, and have other mathematicians rely on it before the mistake is discovered. Papers can be flawed. The papers are not the framework. The reason you think they are the same thing is probably that mathematics has not had that many big mistakes recently, since the renewed effort on solid foundations last century, which means the system is working. You can't define your way out of the fact that mathematicians will make mistakes and write papers that draw conclusions that are not valid. They are just pretty good at discovering the mistakes, and finding them is culturally essential to the practice of mathematics; arguably if it were not, it would be a big game of Numberwang. Psychologists are, on the other hand, not nearly as good at discovering mistakes. We didn't have to use any properties of the self-containedness of the disciplines to get to that result. It was not necessary to prove mathematically that mathematics does not or cannot have a replication problem. You can just observe it. (Edit, all of that in brief: if it were truly impossible to be wrong in maths, then why do mathematicians spend so much time trying to verify each others' proofs? Are they wasting their time? Or is it the entire reason there isn't a replication problem?)
somenewaccount1 5y ago
Here's where you are wrong > But they might! And you would not notice that the framework had become inconsistent for a while. As the author stated, the math can be reproduced by reading it: ergo, thee is no barrier to entry except knowledge. The other sciences do not have this luxury - they require access to specialized instruments, participants, funding, and most importantly - time. Math can be checked nearly instantaneously, so the "might" becomes inconsequential my small.
- FeepingCreature 5y ago
  Also, math is universal. A study can be true in one place and false in another; a proof cannot. This is another barrier to replication.
- kuhewa 5y ago
  > Math can be checked nearly instantaneously, so the "might" becomes inconsequential my small. Can it though? Pickovers 2013 edge response comes to mind, can hundreds of pages of a proof based on axioms that require particular expertise to understand be checked instantaneously? I think it would easier to replicate the typical psychology experiment than to check the math in that case. https://www.edge.org/response-detail/23670 https://www.edge.org/response-detail/23670
  - somenewaccount1 5y ago
    A few edge case examples do not constitute a crisis. The point is not that 0 cases happen in mathematics, but rather that unreproducible results are not the norm, while in other sciences, it is the norm.
    - kuhewa 5y ago
      You might be mistaking me for someone who stated there is a math replication crisis. I replied to one specific claim
- IggleSniggle 5y ago
  > no barrier to entry except knowledge That’s fundamentally it, there’s a barrier to entry, and it’s knowledge. Other sciences may require specialized instruments and time, but math is also subject to the problems of knowledge not being instantaneously transferable. And I imagine, although I’m not certain how prevalent it is, that many branches of mathematics are also dependent on instrumentation these days, in the form of software. And we all know how software can be…
openfuture 5y ago
I think this discussion is a bit off, moving the root of trust from the mathematicians to the computers is the thing that will not only kill mathematics but also our civilization (and I will try to argue this overstatement now). People have a hard time understanding things on extreme ends of the scale because they just assume everything is within some reasonable bounds of effort, this is a failure of imagination. Mathematics is not a reasonable amount of effort. What's more it is depended on by literally everything else we do (as a civilization). Theorem proving software (so far) makes the fatal mistake of just trusting; the machine it is running on, the operating system it is running in, and several other things that can subvert assumptions. The correctness of a proof is justified by the trust we place in the heavily audited kernel which will check the proof script outputted by compilers for the human (textual) interface but that kernel is just another program on your (mostly) proprietary platform. Mathematics is a religion that currently dominates the planet (and as with most religions, it is totally ambient to people in that they don't realize to what extent they base their behaviour on it) but it is not without competition and the thing we must realize is that this competition is as motivated, ruthless and mission-driven as anything else on the planet. Just as we have people sitting around learning to recite the qur'an or the bible from memory we also have universities full of mathematics students reciting proofs and theorems by memory, trying to keep the knowledge that brought us this far alive. Computer memory is inherently volatile and we do not have the same conventions and procedures in place as we do for safeguarding the legitimacy of written texts. These take time to develop and are usually motivated by tragedies (such as the loss of our "prehistory"). Try to imagine the amount of effort it would take to re-assess every piece of information that we have (at current time) because we accidentally let the entropy into our libraries.
- ConceptJunkie 5y ago
  You seem to believe that computer software and hardware has no error detection and correction built into it. This is definitely not true. The reproducibility of computer-derived knowledge is what makes it something we can trust. Especially if it's being reproduced on disparate hardware and disparate software. Mathematics is possibly the only thing in the entire world that can _not_ be described as a religion.

inglor_cz 5y ago

This scenario was actually pretty close when Andrew Wiles published his proof of the Last Fermat Theorem. The proof was huge, IIRC over 100 pages of maths, some of it pretty obscure, and verifying it took a lot of work of several people. They even found a flaw in it, but Wiles was able to fix it after a year.

benibela 5y ago
Or now the "proof" of the abc conjecture with the inter-universal Teichmüller theory

scoofy 5y ago

I’m talking about an event like the incompleteness theorem that ends aspirations like those of Russell and Whitehead. Paradoxes that lead to new axioms like in the development of set theory. Deductive frameworks can’t be wrong in the way that experimental results can be statistical anomalies. The author conflates errors with perfectly reasonable experimental results later shown to be anomalous upon replication.

cormacrelf 5y ago
Again you are talking about the framework as if it is exactly the same thing as the papers that attempt to describe it. It is not. Your statement that "deductive frameworks can't be wrong" is irrelevant to the question of whether a math paper can be wrong. If I prove something, it can be wrong, and it may turn out that I haven't proved it at all, but I can still call it a proof, convince other people it's correct, get it published in a journal, and have other mathematicians rely on it before the mistake is discovered. Papers can be flawed. The papers are not the framework. The reason you think they are the same thing is probably that mathematics has not had that many big mistakes recently, since the renewed effort on solid foundations last century, which means the system is working. You can't define your way out of the fact that mathematicians will make mistakes and write papers that draw conclusions that are not valid. They are just pretty good at discovering the mistakes, and finding them is culturally essential to the practice of mathematics; arguably if it were not, it would be a big game of Numberwang. Psychologists are, on the other hand, not nearly as good at discovering mistakes. We didn't have to use any properties of the self-containedness of the disciplines to get to that result. It was not necessary to prove mathematically that mathematics does not or cannot have a replication problem. You can just observe it. (Edit, all of that in brief: if it were truly impossible to be wrong in maths, then why do mathematicians spend so much time trying to verify each others' proofs? Are they wasting their time? Or is it the entire reason there isn't a replication problem?)

somenewaccount1 5y ago

Here's where you are wrong > But they might! And you would not notice that the framework had become inconsistent for a while. As the author stated, the math can be reproduced by reading it: ergo, thee is no barrier to entry except knowledge. The other sciences do not have this luxury - they require access to specialized instruments, participants, funding, and most importantly - time. Math can be checked nearly instantaneously, so the "might" becomes inconsequential my small.

FeepingCreature 5y ago
Also, math is universal. A study can be true in one place and false in another; a proof cannot. This is another barrier to replication.
kuhewa 5y ago
> Math can be checked nearly instantaneously, so the "might" becomes inconsequential my small. Can it though? Pickovers 2013 edge response comes to mind, can hundreds of pages of a proof based on axioms that require particular expertise to understand be checked instantaneously? I think it would easier to replicate the typical psychology experiment than to check the math in that case. https://www.edge.org/response-detail/23670 https://www.edge.org/response-detail/23670
- somenewaccount1 5y ago
  A few edge case examples do not constitute a crisis. The point is not that 0 cases happen in mathematics, but rather that unreproducible results are not the norm, while in other sciences, it is the norm.
  - kuhewa 5y ago
    You might be mistaking me for someone who stated there is a math replication crisis. I replied to one specific claim
IggleSniggle 5y ago
> no barrier to entry except knowledge That’s fundamentally it, there’s a barrier to entry, and it’s knowledge. Other sciences may require specialized instruments and time, but math is also subject to the problems of knowledge not being instantaneously transferable. And I imagine, although I’m not certain how prevalent it is, that many branches of mathematics are also dependent on instrumentation these days, in the form of software. And we all know how software can be…

openfuture 5y ago

I think this discussion is a bit off, moving the root of trust from the mathematicians to the computers is the thing that will not only kill mathematics but also our civilization (and I will try to argue this overstatement now). People have a hard time understanding things on extreme ends of the scale because they just assume everything is within some reasonable bounds of effort, this is a failure of imagination. Mathematics is not a reasonable amount of effort. What's more it is depended on by literally everything else we do (as a civilization). Theorem proving software (so far) makes the fatal mistake of just trusting; the machine it is running on, the operating system it is running in, and several other things that can subvert assumptions. The correctness of a proof is justified by the trust we place in the heavily audited kernel which will check the proof script outputted by compilers for the human (textual) interface but that kernel is just another program on your (mostly) proprietary platform. Mathematics is a religion that currently dominates the planet (and as with most religions, it is totally ambient to people in that they don't realize to what extent they base their behaviour on it) but it is not without competition and the thing we must realize is that this competition is as motivated, ruthless and mission-driven as anything else on the planet. Just as we have people sitting around learning to recite the qur'an or the bible from memory we also have universities full of mathematics students reciting proofs and theorems by memory, trying to keep the knowledge that brought us this far alive. Computer memory is inherently volatile and we do not have the same conventions and procedures in place as we do for safeguarding the legitimacy of written texts. These take time to develop and are usually motivated by tragedies (such as the loss of our "prehistory"). Try to imagine the amount of effort it would take to re-assess every piece of information that we have (at current time) because we accidentally let the entropy into our libraries.

ConceptJunkie 5y ago
You seem to believe that computer software and hardware has no error detection and correction built into it. This is definitely not true. The reproducibility of computer-derived knowledge is what makes it something we can trust. Especially if it's being reproduced on disparate hardware and disparate software. Mathematics is possibly the only thing in the entire world that can _not_ be described as a religion.