18 ms·

Why isn't there a replication crisis in math?

302 points by jseliger 5y ago

keithalewis 5y ago
"A mathematician's fame is proportional to the number of incorrect proofs they have published."
deltaonefour 5y ago
This gets at the heart of the difference between science and math. What is actually science? and what is actually math? You may think you know, but most people don't. Did you know? That in Science, and therefore reality as we know it... NOTHING can be proven to be true. Proof is the domain of logic and mathematics, NOT science. There is a huge difference between science and mathematics. Math is an imaginary game involving logic. You first recursively assume logic is true then you assume some additional things are true called "axioms." You assume the entire universe only encapsulates Logic and the axioms as part of your theory and that is everything that exists in your theoretical universe. You derive or prove statements that you know must consequently be true in your made up universe based off of logic and the "axioms." These statements are called "theorems." That is math... For example the "pythagorean theorem" is a statement made about geometry assuming that the axioms of geometry are true and logic is true. So of course if it's a logical game, there's no real replication crisis. All theorems in mathematics are proven true based off of logic. You can't replicate conflicting results if a result is PROVEN to be true via pure logic. Science is different. Nothing can ever be proven to be true. There are basically two axioms in science. We assume logic is true, just like in math. And we assume that the mathematical theory of probability is a true model for random events happening in the real world. Then we assume everything else about the universe is completely unknown. This is key. Because the domain of the universe is unknown... at any point in time we can observe something that contradicts our initial hypothesis. ANY point in time. That means even the theory of gravity is forever open to disproof. This is the reason why NOTHING can be proven. To quote Einstein: "No amount of experimentation can ever prove me right; a single experiment can prove me wrong." So sure mathematicians can make mistakes... and I know the author is talking about higher level details.... but at its most fundamental level, assuming all other attributes are ideal... it is fundamentally impossible for math to have a replication crisis while science, on the other hand, is fundamentally forever open to these crisis so long as the universe is unknown. The most interesting thing to me in all of this however is that within science, probability is a random axiom. We have no idea why probability works... it just does. It's the driver behind another fundamental concept: Entropy and the arrow of time. For some strange reason Logic in our universe exists side by side with probability as a fundamental property.
- mcphage 5y ago
  > All theorems in mathematics are proven true based off of logic. You can't replicate conflicting results if a result is PROVEN to be true via pure logic. You can publish a paper in mathematics that claims to prove something, but is mistaken. A paper claiming that a theorem is proven, is not the same thing as the theorem being proven. However, that's not often the case—why that is, is an interesting & meaningful question.
  - deltaonefour 5y ago
    Yeah I get that. What I'm saying is that outside of mathematics, a proof is fundamentally impossible. It can never happen.
- twobitshifter 5y ago
  Even math has the problem you describe. https://en.m.wikipedia.org/wiki/G%C3%B6del's_incompleteness_theorems https://en.m.wikipedia.org/wiki/G%C3%B6del's_incompleteness_...
  - deltaonefour 5y ago
    Yeah, but nobody tries to prove anything that can't logically be proven. So not only is there no replicability crisis but such a thing can't exist in the first place.
    - morelisp 5y ago
      People try all the time to prove things that can't logically be proven. Probably lifetimes spent on Euclid's parallel postulate alone - over 2000 years of attempts! Before Russell Frege thought his logic was consistent. Before Gödel it was believed we could formalize a complete arithmetic even if Frege's specific design failed. Before Church and Turing the Entscheidungsproblem was considered potentially solvable even if Gödel's work ruled out the resultant proofs wholly capturing powerful theories. It took a very long time to even notice the axiom of choice, and longer to ultimately prove its proof futile.
deleted 5y ago
[deleted]
tabtab 5y ago
because practical applications of such works are usually several decades away. A bad medical study can kill people today. There's not a lot of incentive to check nor complain about consequences far into the future.
Iwan-Zotow 5y ago
Huh?!? Please read about abc conjecture and whole saga wrt proof, Shinichi Mochizuki, Peter Scholze, etc, etc, etc
- sgillen 5y ago
  Did you read the article? This saga is explicitly mentioned and does not detract from the authors point IMO.
  - Iwan-Zotow 5y ago
    And everything (absolutely EVERYTHING) is open and published
  - Iwan-Zotow 5y ago
    But it does! Author is just plain wrong. Here there is piece of the very important math research which community failed to replicated in either positive way (confirm it) or negative way (reject it)
    - sgillen 5y ago
      Yes but it's famous because it's the exception to the rule. It's definitely a failure to replicate, but that does not necessarily mean that math has a replication crises in the same way the social sciences do.
      - Iwan-Zotow 5y ago
        > It's definitely a failure to replicate, but that does not necessarily mean that math has a replication crises in the same way the social sciences do. But it does! Look, in social sciences there are a lot of lousy papers written by people who doesn't know statistics, acceptance criteria etc. There are problems with "average" papers. In math there are no problems with "average" papers, afaik. People are following those papers and rather quickly find flows or confirm the result. In math there are problems with "exceptional" papers. Holes in Fermat theorem proof or abc paper were taken years to find, patch and settle.
otabdeveloper4 5y ago
Because math is not a science.
- iqanq 5y ago
  More like because maths are an actual science.
mjfl 5y ago
because math doesn't do experiments.
- deleted 5y ago
  [deleted]
- NovemberWhiskey 5y ago
  This seems the obvious answer. The replication crisis isn't about published material being wrong, it's about the inability to reproduce the results of experiments or studies in a repeatable fashion. It's not like you make a hypothesis in math and then need to go away and interview a sample of 1,000 circles and report back that, controlling for ellipses that may be misreporting as circles, the ratio of the circumference to the diameter is 3.2 +/- 0.1 (p<0.05).
- medstrom 5y ago
  Exactly. A math study is not a "study" in the sense of "hey i saw a funny pattern in some data maybe it's a sign of my pet theory" - it's literally already proven when published. There's nothing more to do.
  - morelisp 5y ago
    There are a lot of math papers / work that are "hey I saw a funny equation/morphism/shape/etc, what does it 'do'?" that present a lot of conjectures / constructions but are weak on any conventionally called a theorem. https://erich-friedman.github.io/packing/ https://erich-friedman.github.io/packing/ is probably one of the best known on HN.
- Jtsummers 5y ago
  Or as noted in the article: > But one of the distinctive things about math is that our papers aren’t just records of experiments we did elsewhere. In experimental sciences, the experiment is the “real work” and the paper is just a description of it. But in math, the paper, itself, is the “real work”. And > And that means that you can replicate a math paper by reading it.
the_watcher 5y ago
The social science and medical replication crisis seems like it would be far more impactful than a mathematics crisis, right? Politicians, policy-makers, doctors, etc. all make decisions based on potentially flawed or outright incorrect studies in a way that I don't think is true for the equivalents in math, simply because there aren't decisions and policies up for debate related to much of them (if I am wrong about this, please correct me).
- smartscience 5y ago
  If a flawed mathematical paper were used as the basis for what then became a flawed cryptography algorithm, I can see that having impact if the bad guys noticed the flaw first. But yes, I expect examples like that would be comparatively rare.
  - the_watcher 5y ago
    Thanks for this, cryptography is a good example where this could be a problem.
  - not2b 5y ago
    In cryptography the math is almost always the strongest part, and it is the side-channel attacks and implementation mistakes that let the bad guys in. When it is the math, the flaw is often that the algorithm has all the desirable properties proved in a number of papers, but has some exploitable structure that analysts can turn into an attack.
- IshKebab 5y ago
  I think politicians generally don't make policy decisions based on science. I live in the UK and an obvious example is drug policy. They even fired the scientist whose drug research they didn't like. But if the science agrees with a decision they've already made then they're happy to use if for justification, even if the science is junk (e.g. the crazy fines on taking children out of school in the UK).
gus_massa 5y ago
> When I’ve served as a peer reviewer I’ve read the papers closely and checked all the steps of the proofs, and that means that I have replicated the results. The side effect is that math papers have an insane long time to publication. Perhaps 6 months, or 1 year or more if you are unlucky. In physics, the publication time is like 3 month. Something like 1 month for the first review and then two months for making small changes suggested by the referee and discussing with the editor. As a side^2 effect, some citation index of the journals count only the citations during the first year. But the papers that have the citation are sleeping over the reviewer desk during that year, so the number is lower than the real number.
Barrin92 5y ago
why not turn the premise from the article around. Instead of suggesting that math might also have a replication crisis, why not question whether the whole replication crisis thing was overblown and effectively more of an ideological attack on a few disciplines. Errors in science, disagreement and lack of reproducibility I think are common and prevalent but it doesn't necessarily imply that a discipline as a whole doesn't make progress. The obsession with statistical accuracy and 'science as bookkeeping' mentality seems fairly new to begin with, and science did just fine before we even had the means to verify every single thing ever published. It kind if ignores the dynamic nature of science. Most of what is published probably has close to zero impact regardless of whether its right or wrong, but paradigm changing research generally asserts itself. Science is evolutionary in that sense, it's full of mistakes but stumbles towards correct solutions at uneven tempo. In a sense you can just look at it like VC investment. Nine times out of ten individual things don't work, but the sector overall works, the market economy is full of grifters and failed businesses, but it doesn't matter that much. So, maybe half of math is bullshit but so is everything else but in math people just say "whatever" until they find something good, whereas in psychology people use it as an opportunity to hack away at it.
- burnished 5y ago
  This isn't turning the problem around in the same sense. In order to effectively turn a problem around you need to use its complement, and then it should be a binary proposition. Your example where you suppose an idealogical attack fails here because it is not the only other explanation, you haven't turned the problem around in the same way that turning "what is the probability this action had an effect" around can become "what is the probability that this action had no effect".
- quanticle 5y ago
  It kind if ignores the dynamic nature of science. Most of what is published probably has close to zero impact regardless of whether its right or wrong, but paradigm changing research generally asserts itself. The problem is that psychology, unlike physics, doesn't really have a paradigm. There's the old saying, "Extraordinary claims require extraordinary evidence." But for that heuristic to be effective, you need a standard for what counts as extraordinary. Extraordinary compared to what? In phsyics, there are two well-established paradigms (relativity and quantum mechanics), which establish what counts as ordinary, and what counts as extraordinary. So, for example, if you're making a claim that the distribution of dark matter in the cosmos more clumpy than predicted by existing models, of that the energy level of a particular field is 12 MeV rather than 10, those are ordinary claims, which can be accomodated by tweaks to the existing paradigm. But if you're saying that the speed of light has varied over the history of the universe, or that all subatomic particles are actually tiny vibrating string-like structures, well, that's going to require a lot more evidence. In psychology, it's much more difficult to have that kind of intuition. Take the concept of priming, for example. Is claiming that people walk more slowly when they're encouraged to think of things that make them feel old extraordinary? It makes a certain sort of intuitive sense, but, on the other hand, there's absolutely no causal mechanism suggested. So when a number of priming studies fail spectacularly under replication [1], I don't know what to think. I don't have a good sense for how much of psychology is overturned by the replication failure, in the same sense that I'd have for physics if it turned out that e.g. the speed of light is a variable rather than a constant. [1]: https://mindhacks.com/2017/02/16/how-replicable-are-the-social-priming-studies-in-thinking-fast-and-slow/ https://mindhacks.com/2017/02/16/how-replicable-are-the-soci...
svat 5y ago
Despite the title (a title with a question in it invites people to comment without reading the post, even more than the usual already high level), this is a really good post IMO, with valuable insights into not just mathematics but also the replication crisis elsewhere. (And it does discuss Mochizuki's claimed proof of the abc conjecture, and links to MathOverflow on formalization with proof assistants, to a recent paper discussing Vladimir Voevodsky's views, etc.) This from the first part is sound: > The replication crisis is partly the discovery that many major social science results do not replicate. But it’s also the discovery that we hadn’t been trying to replicate them, and we really should have been. In the social sciences we fooled ourselves into thinking our foundation was stronger than it was, by never testing it. But in math we couldn’t avoid testing it. But the post doesn't stop there! The second part of the post (effect sizes etc), with the examples of power posing and the impossibly hungry judges, is even more illuminating. Thanks!
zitterbewegung 5y ago
Math doesn’t have a replication crisis at all it has a comprehension crisis. Since proofs are programs one can basically say that mathematical theorems are incredibly detailed software that is completely open source and invites people to identify programs that don’t work and or fix issues. A famous one is Fermats last theorem which needed a fix but was largely right. Others have said that it takes 6 months to a year to get published. The other thing with math is the fact that you can get completely scooped and your work is worthless. Edit: I am using "proofs are programs" very loosely and yes Theorems are much more than programs as other commenters have pointed out.
- contravariant 5y ago
  A more notorious example of the comprehension crisis would be Mochizuki's claimed proof of the abc conjecture. So far fairly few people are willing to claim they both understand and agree with the several hundred pages of 'proof'.
  - exdsq 5y ago
    https://en.wikipedia.org/wiki/Shinichi_Mochizuki https://en.wikipedia.org/wiki/Shinichi_Mochizuki It's a fun rabbit hole to go down :)
  - zitterbewegung 5y ago
    I was tempted to use that but Fermat's last theorem is known to the general public for much longer and has a resolution.
    - contravariant 5y ago
      It was a good example, and definitely one of the more important examples of how finding and fixing mistakes in mathematics is supposed to happen. I figured the abc-conjecture would provide a nice contrasting example.
- Quekid5 5y ago
  An interesting perspective on programs-as-proof is I-forget-his-name-but... the mathematician who made really bold claims, if only you'd study under his tutelage for a number of years to understand this whole new terminology he invented. With programs-as-proof it really wouldn't matter. It's either "computer says yes" or "compu'er says noooo". EDIT: Whoop, sibling post mentioned, it's Mochizuki.
Simplicitas 5y ago
Steven Pinker takes a decent crack at "Statistical Significance" in his new book Rationality (https://en.wikipedia.org/wiki/Rationality_(book) https://en.wikipedia.org/wiki/Rationality_(book)), which underpins a lot of this and is mentioned in this piece. And I'm still grappling with this part of the book. lol
mjw1007 5y ago
I think it's a good article. But it's worth considering the possibility that mathematics could have fallen, and could still fall, into a state where false results are frequently published, and isn't 'protected' by anything special in the nature of the field or its practitioners. Just as you might find yourself asking "why did city A fall into the grip of organised crime, and not city B?". You might look for answers in the methods of police recruitment or a strong history of respect for the rule of law or anything like that, but it might turn out that the answer is really just "city A got unlucky".
raphlinus 5y ago
This is a thoughtful and thought-provoking blog post. I think it's worth asking similar questions of computer science. I think you'll find some math-like patterns -- there's basically no chance Quicksort or any other fundamental algorithm paper will fail to replicate -- and some patterns which will fail to replicate, like in software engineering. Some of the early results on pseudorandom generators and hash functions aren't holding up well, but I think that's just progress. We understand the problem a whole lot better than we did back then. Perhaps more interesting is the literature on memory models. The original publications of the Java and C11 memory models had lots of flaws, which took many years to fix (and that process might not be totally done). I worry that there are a bunch of published results that are similarly flawed but just haven't gotten as much scrutiny.
- ahelwer 5y ago
  In competitive programming you could basically assume the pseudocode in a paper is not literally correct and requires some tweaking to work, despite a “proof” of its correctness. Particularly with string algorithms.
  - sterlind 5y ago
    long time no see! there's a couple levels there: rote translating pseudocode into your target language isn't likely to pan out well. so instead you run the pseudocode in your mind, develop an intuition on how it works, and that's the "replication" bit this post talks about with reviewing math papers. but both the pseudocode and your code will likely have edge cases you didn't handle. this isn't a problem for math - that's the category of common trivial/easily fixable proof errors that don't really affect the paper. but they're a problem for machines that run them literally. maybe a good compromise strategy for formal verification is to declare the insight of the algorithm - recurrence relation or whatever - as an axiom, and then use the prover to whack the tricky edge cases.
- Analemma_ 5y ago
  The parts of CS that are the most math-like (which include fundamental algorithms) don't have a replication crisis, but the ones that are the most social-science like probably do, or would. I would bet large sums of money that a lot of the literature on stuff like "does OOP lead to better code", "does IDE syntax highlighting lead to fewer bugs" etc. would fail to replicate if anyone bothered trying. The thing is, the general sense I get is that people in CS already have so little confidence in these results that it's not even considered worth the time to try and refute them. Which doesn't exactly speak well of the field!
schuyler2d 5y ago
You could also argue that Math had its replication crisis in the 17th-19th centuries. E.g. infinite series "proofs" that were eventually shown to be flawed methodologies. This and other crises led to grounding modern mathematics with set theory, Zermelo–Fraenkel axioms, etc and understanding what's possible (e.g. Godel's theorem). Psychology and other social sciences are barely a century old.
- fshbbdssbbgdd 5y ago
  A more recent example is the Italian school of algebraic geometry, where it was discovered in the mid 20th century that many claimed proofs were faulty (and some “proven” results were incorrect).
- zmgsabst 5y ago
  Mathematics is having a replication crisis and people pay so little attention they don’t know. That replication crisis has led to efforts in formal verification such as HoTT, Lean, etc. https://homotopytypetheory.org/ https://homotopytypetheory.org/ https://xenaproject.wordpress.com/2021/06/05/half-a-year-of-the-liquid-tensor-experiment-amazing-developments/ https://xenaproject.wordpress.com/2021/06/05/half-a-year-of-...
  - Victerius 5y ago
    How is math experiencing a replication crisis? > This site serves to collect and disseminate research, resources, and tools for the investigation of homotopy type theory, and hosts a blog for those involved in its study. > Exactly half a year ago I wrote the Liquid Tensor Experiment blog post, challenging the formalization of a difficult foundational theorem from my Analytic Geometry lecture notes on joint work with Dustin Clausen. ???
  - dwohnitmok 5y ago
    No by and large mathematics is not having a replication crisis. As the blog post in your link states: > Question: Was the proof in [Analytic] found to be correct? > Answer: Yes, up to some usual slight imprecisions. This has been the case for almost all math formalization efforts. Even when (very rarely) proofs were revealed to be incorrect, the result was salvageable.
dilawar 5y ago
A bit cynical take is that it is harder to mislead in theory? Your mistakes will get caught and you risk losing your reputation so it less likely that someone will publish a proof without double triple checking. Theory subject has faster verificarion loops. Verification and review is harder in experimental works and it might take few decades before someone finds an error, or even collect resources to verify the claim. So maybe it's harder to fight 'publish or perish' deamons in experimental sciences?
verisimi 5y ago
"We get away with it becuase we can be right for the wrong reasons—we mostly only try to prove things that are basically true." Apart from the replication crisis, the other crisis that is not really talked about is funding. Academic funding basically comes from one of 3 (connected) sources - government, corporations and the military. Somehow or other - these 3 sources have pretty much the same or non-conflicting aims. These aims relate to power and control. This is actually the largest crisis, IMO. Given that information, we can re-assess the quoted statement the author makes. Perhaps its not that they are proving things that are "basically true". Its that right or wrong do not matter. What does matter is that the answers provided meet the agenda of those funding the study. The answer is not that important as long as it is supportive of whatever agenda is in play. I believe this is the case for the replication crisis in science also. A replication "crisis" is only a crisis if you are attempting to achieve truth and greater understanding. But truth and understanding are only ostensible reasons, not the actual ones. What these studies are actually doing is creating a parallel construction - the aim is actually for studies to appear 'truthey', without actually being so. What studies should actual do is increase the funder's power, wealth extraction abilities, etc. If you doubt this and think that truth matters, consider this. Surely we should have cracked the best diet for people by now? But there is no common understanding of what is good or bad to eat - if anything there is more confusion. The reason of course, is that there's no money in recommending whole foods or whatever. However, there is money in drugs to make people 'better'. And money in making diet so confusing that people eat themselves into trouble. Anyway, if you are in the business of governing or monetising the masses, truth and understanding is the last thing you want. Far better to have a story that gives you control, or extracts money. Such is life under fascist governance (where fascist = corporation + governance working together).
scoofy 5y ago
The paper here seems to make absolutely zero distinction between deductive and inductive reasoning, which should be the entire point. Math is an arbitrary framework, built from arbitrary axioms. It is deductive. Thus, all proofs are simply deduction. The knowledge here is positive knowledge, we can show things are true, false, or undecidable. There may be errors, but those are errors in execution. Psychology is not built on a framework. It is inductive, we are trying to find axioms that map to the data we collect. Thus, all papers are trying to add/build it's arbitrary framework. The only knowledge here is negative knowledge, falsification, we know only what is a failed hypothesis. There will be errors, both in execution, and there will also be statistical errors in experimental results. The entire point of the replication crisis is that we don't publish or pay attention to results that are boring, so the framework we build is built on skewed data. We don't reject previously popular papers that are now unfalsifiable (the idea that the now unfalsifiable Milgram experiment is still taught in every university psychology dept should be outrageous). The boring results need to be weight statistically against the interesting results, but aren't, etc. Nobody out there is arguing whether or not the axiom of choice is a true axiom? It sort of doesn't matter, it can't matter, because it's arbitrary by definition. You can't have a replication crisis inside of a deductive framework without changing the framework. This doesn't happen too often, but we did see this during the shift from neutonian to einsteinian physics. The study of philosophy of science is fairly obscure, but is the center of this discussion.
- netizen-936824 5y ago
  I don't believe I'm familiar with an Ingram experiment, any chance you could expand on this? I was also unable to find something with a quick internet search, however Milgram was suggested to me.
  - scoofy 5y ago
    Milgram experiment, sorry... must have got hit by autocorrect there: https://en.wikipedia.org/wiki/Milgram_experiment https://en.wikipedia.org/wiki/Milgram_experiment You can add the Stanford Prison Experiment (Zimbardo) to the list of, now unfalsifiable, experiments that inexplicably are regularly taught in university settings when they cannot possibly provide any useful data to the sciences: https://en.wikipedia.org/wiki/Stanford_prison_experiment https://en.wikipedia.org/wiki/Stanford_prison_experiment The idea that many if not most universities create educational programs based on these very obvious problematic studies, leads me to believe that the scientific community can be as guilty of info-tainment bias as the general public. I only wish that I'd been able to continue in academia so that i could have at least a small voice in changing this problematic dynamic we live with. They literally made a movie about the Zimbardo experiment in 2015. It couldn't be more obvious that people care more that it be true than whether it might be false: https://en.wikipedia.org/wiki/The_Stanford_Prison_Experiment_(film) https://en.wikipedia.org/wiki/The_Stanford_Prison_Experiment...
curt15 5y ago
It did: https://en.wikipedia.org/wiki/Italian_school_of_algebraic_geometry https://en.wikipedia.org/wiki/Italian_school_of_algebraic_ge...
ocschwar 5y ago
There is one corner of math that does have a replication crisis. Just as we compare programming languages by how "ergonomic" they are to learn and use, mathematicians do come up with novel notation systems to try to improve the ergonomic state of their field, and since "ergonomics" is another way to say "esthetics", and is proved or disproved by user testing, that is where replication gets hard. The inventor of category theory's wiring diagrams, for example, has claimed that he could get middle schoolers to understand them. I suspect that success has not been replicated.
- mlyle 5y ago
  > The inventor of category theory's wiring diagrams, for example, has claimed that he could get middle schoolers to understand them. I suspect that success has not been replicated. Maybe I should try. I like category theory and I am a middle school teacher and like teaching electives on improbable things :D
lacker 5y ago
The root of the replication crisis in social sciences is not just that many papers fail to replicate, but that there is no way to clearly resolve a result that fails to replicate. A paper claims that pre-K increases test scores a decade later, another paper claims it doesn't, and there's no clear resolution. The disagreement just festers, with both sides citing research that supports their opinion. The argument often "spills out" into the public sphere. In mathematics and computer science, there are many errors in published papers. However, once you point out an error, it's usually pretty straightforward to resolve whether it's really an error or not. Often there is a small error which can be fixed in a small way. Exceptions like the abc conjecture are rare.
umvi 5y ago
My completely subjective opinion is that at the highest levels of math, there are only a handful of people that are even capable of peer reviewing, and their time is in high demand. Wiles's proof of Fermat's Last Theorem is like 120 pages long and he first delivered it disguised as a class to a bunch of grad students who barely understood any of it and hence gave no feedback. Because this is Fermat's Last Theorem which is famous, eventually people in the math community that understood Wiles's work reviewed it and found an error. Had it been a 120 page proof of some not famous problem like random chessboard thought experiments, it probably could go years without anyone seriously looking at it.
amelius 5y ago
Because math is not science? https://philosophy.stackexchange.com/questions/14818/is-mathematics-considered-a-science https://philosophy.stackexchange.com/questions/14818/is-math...
- kkoncevicius 5y ago
  I wanted to post the same thing, so was looking for a top comment addressing it and glad to have found it. It's weird how many people, even on HN, do not get this distinction. Math gets thrown in with all the sciences but it itself is not a science - it doesn't have experiments, it doesn't follow the scientific method. Mathematics cannot have a replication crisis because there is nothing to replicate. In math correctness is tested not by redoing something but by double-checking the original work for mistakes.
unemphysbro 5y ago
2+2=4
jll29 5y ago
Mathematics is fully deductive (so it is not strictly speaking a science - it is not empirical, but it's own kind of discipline, like logics, if you don't consider that part of mathematics), so replication can be attempted by machine, and as the authors wrote proof-checkers find more and more mistakes. A water-tight proof essentially is the trace or log showing a series of mechanical deductive steps. But because journals publish human proofs, one of the following cases can arise: - A proof can be wrong and the underlying conjecture can still be true (i.e. a theorem). - A proof can be wrong and the conjecture can be false - these kinds of errors need to be corrected with utmost urgency, because they lead to follow-up downstream errors. - A proof can be right and recognized as such by peers, in which case it gets published and everybody is happy. - A proof can also be right and be contested by peers. This is when proofs have gaps that are "obvious" to some, but not believed by others. Because a proof is a sequence of steps that are either previously derived or self-evident, and people simply differ what they consider self-evident (as an aside, in school I was criticized in maths for "skipping steps" and in English for "jumping thoughts"; to me it seemed obvious where things were going but the teachers obviously needed "comments to the code", so I learned to insert baby steps, and everyone was happy). Because of the last case, it is important to get down to the most formal, fine-grained, nitpicky, atomic level, so everyone can agree that each micro-step is self-evident, and thankfully we can dedicate this to machines nowadays (provers like the "Isabelle" system - https://www.cl.cam.ac.uk/research/hvg/Isabelle/ https://www.cl.cam.ac.uk/research/hvg/Isabelle/), at least to an extent. Ironically, that's not how real proofs in mathematics journals look like at all. They're written in prose, and often written in a surprisingly "meta" style (we had algebra lectures where the professor talked about "colored roosters" and how they behave, which is the name of some structure in the algebraic sub-field of Ramsey theory).
graderjs 5y ago
Because it's largely not an experimental science, but rather is groups of people convincing themselves they know what someone is talking about. As the article pretty much says. The article also suggests an "exceptionalism" of maths in that their results hold up even when their methods don't. In other words math papers are full of "unimportant flaws and errors" but even though that's the case when corrected or verified, somehow the conclusions of the papers mostly hold up, in the author's opinion. But I tend to think this is more because, math is groups of people (and formal verification systems) convincing themselves they know what someone else is talking about, so confirmation bias is probably in effect. The limitations of a self-consistent set of axioms after all... You may say that formal verification systems are perfect oracles... But if papers have "unimportant flaws and errors" it's possible that those systems have them, too, right? It's also possible that as those systems were written by people, they reflect and somehow seek to confirm the biases of those people or their understandings (or misunderstandings) about mathematical knowledge. So I don't think maths is quite as exceptional as this author hopes it might. Tho Maybe it is
xiphias2 5y ago
There are lots of cryptographical protocols with mathematical proofs that contain errors that break the security of the protocols themselves. A version MuSig in Bitcoin was a protocol like that, though luckily the mistake was caught by the cryptographic community when the authors wanted to publish the proof at a conference.
topaz0 5y ago
It seems that a lot of commenters here have not read the post, and are simply posting their own opinions about how to answer the question in the title. If that is you, and you are interested in the question, I recommend reading it in full -- there are a lot of interesting points there.
chrisbrandow 5y ago
when I was getting my PhD in chemistry, the replication crisis had not been recognized (widely anyway). as it has become more widely discussed I've vacillated between annoyed that people don't make the distinction between the hard/simplef sciences and the fields that study much more complicated, multi-casual topics and being self critical that perhaps chemistry also suffers from a widespread problem. I still don't know the answer but I suspect there is something to the simple/complicated distinction. I do recall that there was this one professor that couldn't replicate his seminal work that basically got him tenure and it was all very dramatic. the kind of story that we would share with each other as a byword. Aslo, I wonder if the errors get caught more quickly because in chemistry and physics to a degree, most progress is built directly on previous results. so you can't have bad results just floating unnoticed for very long.
- mateo1 5y ago
  The thing is, mistakes will always be made and corrected. Maybe the experiment worked the first time because someone smoked in the room yesterday and a trace amount of contaminant made it in the solution and catalysed a reaction. Maybe your supplier had a minor contamination on your reactants. The reasons are endless. The issues I have with the "replication crisis" are two: malicious publishing of false results for personal or organisational profit, and malicious interpretation of prestigious publications as the one and only immutable truth. Both of which are social and political problems. Scientific publishing was functioning fine so far for the same reason the old internet was full of true information while it didn't have to: there was little to be gained by lying. Now that gov policies are build on phd papers, things are changing.
pmjones 5y ago
"All science is either physics, or stamp collecting." (Ernest Rutherford)
aaron695 5y ago
mathteddybear 5y ago
One time when I was still a sophomore a friendly professor shared a story how one of his colleagues had a math paper accepted, and found an error in the proof only after it was published. So he had another paper in that journal next, with the refutation of his earlier work. And that feat was worth double the point score awarded to the institute by the scientific powers that be. I'm sure the excuse is that errors in math papers these days don't have the same kind of impact as mistakes in, let's say, medicine. Later on my guess was that this will change when we will have a editor software that is easy to use, wysiwyg, renders math texts as well as latex, and actually the edited document is integrated with the formal proof, beneath the fancy elegant phrases and formulas. Then I quit a job and enrolled for phd studies but that's another story.
dragonwriter 5y ago
Because math isn't science, so there is nothing to replicate. If the analysis is sound, that's it, there's no question of whether the physical results the analysis was applied to are fraudulent, measurement error, due to improper setup, etc. Why is this even a question? Analytical and empirical fields face fundamentally dissimilar challenges.
throwawayarnty 5y ago
The replication crisis is not largely due to the math/stats used in studies. It comes from trying to make inference from noisy and often confounded experiments from limited data.
robertlagrant 5y ago
The article's linked article on social priming has a fun typo: 4.10 The “Lacy Macbeth Effect”
_xnmw 5y ago
Unrelated, but the replication crisis in medicine is the primary reason why anti-vaxxers have a right to be skeptical, even if they're wrong and spout nonsense. Now we *know* that a large percentage of cited medical research is fraught with uncertainty and is potentially non-replicable. Therefore, people have a right to be skeptical of "the science" even if they don't have a coherent counterargument. It is extremely reasonable to be suspicious of statistical methodologies in vaccine safety studies and adverse reaction reporting; there are statistical errors and even outright fraud all over the scientific landscape, so why not here? And given the extremely perverse financial and political incentives found in this particular pandemic environment, I would be extremely surprised if there was NO foul play happening, because you would have to be a hero to ignore these incentives.
bongoman37 5y ago
srcreigh 5y ago
My summary: in social science, measurements are coarse which means the statistics cannot evaluate subtle changes. In math, logic is very precise so subtle changes can be measured. The author makes a point that both social science people and mathematicians are trying to prove subtle things which are actually probably true. In other words the social science replication crisis is because the experiments are often impossible to perform consistently when the effect is subtle, leading to the use of inconsistent lucky draws to demonstrate things.
deleted 5y ago
[deleted]
JPLeRouzic 5y ago
I like this sentence from the link: > Many papers have errors, yes—but our major results generally hold up, even when the intermediate steps are wrong!
croes 5y ago
In mathematics there is no measurement uncertainty
qwerty456127 5y ago
How can there be errors in math papers if math formulae are trivial to check formally using a computer?
- dagw 5y ago
  Because the current state of the art of automated proof checking is still trying to check all the proofs you'll see in a undergraduate program and nowhere near able to check state of the art proofs.
- asdf_snar 5y ago
  Math formulae are not trivial to check formally using a computer. Mathematical software is in its infancy relative to the sophistication of mathematical objects defined in the literature.
benreesman 5y ago
Not being a formally trained mathematician or psychologist but being someone who’s seen plenty of organization/institutional mechanism design: is it possible that e.g.psychology research might become more mathematically rigorous if there were more incentives for serious mathematics people to study psychology? I appreciate at first glance that might seem a bit vacuous: but there’s a ton of money and way better policy at stake if we understood behavioral economics better, and that’s at least an adjacent field to psychology right?
IvanK_net 5y ago
Just a fun fact: there are dozens of papers released each year, which claim that they have proven that P=NP, or P is not equal NP. E.g. they "prove" it with soap bubbles: https://arxiv.org/pdf/cs/0406056.pdf https://arxiv.org/pdf/cs/0406056.pdf
- pfortuny 5y ago
  Well to be honest, things on arxiv are not properly speaking “papers” in the academic sense. At least in Maths.
nemo44x 5y ago
In Soviet Russia, mathematicians, physicists, etc came to the same conclusions as their enemies in the USA. However, the social science results, although unified in each country, were almost always different and in conflict. Social sciences are politics and nothing more. It’s an opinion of how the world ought to be.
ContrarianBrit 5y ago
Very interesting! As a social scientist, the replication crisis gets me seriously down, making me doubt the integrity of my field. It's good to see that there's not such a problem in maths...
LatteLazy 5y ago
Replication crisis come down to funding. If you publish bad results in maths, you lose funding and get fired and humiliated. If you publish bad (but exciting or popular) results in social sciences, you get more funding and acclaim. All subjects are basically (sadly) products. Maths, Physics and Engineering are products that have to (mostly, eventually) actually work. Politics, Sociology, Gender Studies etc are more about entertainment. Som fields (eg nutrition) are a hybred of the two: some serious factual research, some fads and fashion.
phkahler 5y ago
Because math doesnt rely on experiments.
deleted 5y ago
[deleted]
maliker 5y ago
Maybe there isn’t a math replication crisis, but there is a kind of math fragmentation crisis. Sub-fields of mathematics have gotten so deep that specialists can’t communicate with one another [0]. I’ve heard this expressed by other mathematicians as having only 2 or 3 other people in the world that they can discuss their work with. The level of solitary study necessary to make progress is another sign of this phenomenon. Andrew Wiles famously spent six years alone in his attic to come up with his proofs for Fermat’s Last Theorem. [0] https://sites.math.rutgers.edu/~zeilberg/Opinion104.html https://sites.math.rutgers.edu/~zeilberg/Opinion104.html

302 points by jseliger 5y ago

"A mathematician's fame is proportional to the number of incorrect proofs they have published."

This gets at the heart of the difference between science and math. What is actually science? and what is actually math? You may think you know, but most people don't. Did you know? That in Science, and therefore reality as we know it... NOTHING can be proven to be true. Proof is the domain of logic and mathematics, NOT science. There is a huge difference between science and mathematics. Math is an imaginary game involving logic. You first recursively assume logic is true then you assume some additional things are true called "axioms." You assume the entire universe only encapsulates Logic and the axioms as part of your theory and that is everything that exists in your theoretical universe. You derive or prove statements that you know must consequently be true in your made up universe based off of logic and the "axioms." These statements are called "theorems." That is math... For example the "pythagorean theorem" is a statement made about geometry assuming that the axioms of geometry are true and logic is true. So of course if it's a logical game, there's no real replication crisis. All theorems in mathematics are proven true based off of logic. You can't replicate conflicting results if a result is PROVEN to be true via pure logic. Science is different. Nothing can ever be proven to be true. There are basically two axioms in science. We assume logic is true, just like in math. And we assume that the mathematical theory of probability is a true model for random events happening in the real world. Then we assume everything else about the universe is completely unknown. This is key. Because the domain of the universe is unknown... at any point in time we can observe something that contradicts our initial hypothesis. ANY point in time. That means even the theory of gravity is forever open to disproof. This is the reason why NOTHING can be proven. To quote Einstein: "No amount of experimentation can ever prove me right; a single experiment can prove me wrong." So sure mathematicians can make mistakes... and I know the author is talking about higher level details.... but at its most fundamental level, assuming all other attributes are ideal... it is fundamentally impossible for math to have a replication crisis while science, on the other hand, is fundamentally forever open to these crisis so long as the universe is unknown. The most interesting thing to me in all of this however is that within science, probability is a random axiom. We have no idea why probability works... it just does. It's the driver behind another fundamental concept: Entropy and the arrow of time. For some strange reason Logic in our universe exists side by side with probability as a fundamental property.

mcphage 5y ago
> All theorems in mathematics are proven true based off of logic. You can't replicate conflicting results if a result is PROVEN to be true via pure logic. You can publish a paper in mathematics that claims to prove something, but is mistaken. A paper claiming that a theorem is proven, is not the same thing as the theorem being proven. However, that's not often the case—why that is, is an interesting & meaningful question.
- deltaonefour 5y ago
  Yeah I get that. What I'm saying is that outside of mathematics, a proof is fundamentally impossible. It can never happen.
twobitshifter 5y ago
Even math has the problem you describe. https://en.m.wikipedia.org/wiki/G%C3%B6del's_incompleteness_theorems https://en.m.wikipedia.org/wiki/G%C3%B6del's_incompleteness_...
- deltaonefour 5y ago
  Yeah, but nobody tries to prove anything that can't logically be proven. So not only is there no replicability crisis but such a thing can't exist in the first place.
  - morelisp 5y ago
    People try all the time to prove things that can't logically be proven. Probably lifetimes spent on Euclid's parallel postulate alone - over 2000 years of attempts! Before Russell Frege thought his logic was consistent. Before Gödel it was believed we could formalize a complete arithmetic even if Frege's specific design failed. Before Church and Turing the Entscheidungsproblem was considered potentially solvable even if Gödel's work ruled out the resultant proofs wholly capturing powerful theories. It took a very long time to even notice the axiom of choice, and longer to ultimately prove its proof futile.

deleted 5y ago

[deleted]

tabtab 5y ago

because practical applications of such works are usually several decades away. A bad medical study can kill people today. There's not a lot of incentive to check nor complain about consequences far into the future.

Iwan-Zotow 5y ago

Huh?!? Please read about abc conjecture and whole saga wrt proof, Shinichi Mochizuki, Peter Scholze, etc, etc, etc

sgillen 5y ago
Did you read the article? This saga is explicitly mentioned and does not detract from the authors point IMO.
- Iwan-Zotow 5y ago
  And everything (absolutely EVERYTHING) is open and published
- Iwan-Zotow 5y ago
  But it does! Author is just plain wrong. Here there is piece of the very important math research which community failed to replicated in either positive way (confirm it) or negative way (reject it)
  - sgillen 5y ago
    Yes but it's famous because it's the exception to the rule. It's definitely a failure to replicate, but that does not necessarily mean that math has a replication crises in the same way the social sciences do.
    - Iwan-Zotow 5y ago
      > It's definitely a failure to replicate, but that does not necessarily mean that math has a replication crises in the same way the social sciences do. But it does! Look, in social sciences there are a lot of lousy papers written by people who doesn't know statistics, acceptance criteria etc. There are problems with "average" papers. In math there are no problems with "average" papers, afaik. People are following those papers and rather quickly find flows or confirm the result. In math there are problems with "exceptional" papers. Holes in Fermat theorem proof or abc paper were taken years to find, patch and settle.

otabdeveloper4 5y ago

Because math is not a science.

iqanq 5y ago
More like because maths are an actual science.

mjfl 5y ago

because math doesn't do experiments.

deleted 5y ago
[deleted]
NovemberWhiskey 5y ago
This seems the obvious answer. The replication crisis isn't about published material being wrong, it's about the inability to reproduce the results of experiments or studies in a repeatable fashion. It's not like you make a hypothesis in math and then need to go away and interview a sample of 1,000 circles and report back that, controlling for ellipses that may be misreporting as circles, the ratio of the circumference to the diameter is 3.2 +/- 0.1 (p<0.05).
medstrom 5y ago
Exactly. A math study is not a "study" in the sense of "hey i saw a funny pattern in some data maybe it's a sign of my pet theory" - it's literally already proven when published. There's nothing more to do.
- morelisp 5y ago
  There are a lot of math papers / work that are "hey I saw a funny equation/morphism/shape/etc, what does it 'do'?" that present a lot of conjectures / constructions but are weak on any conventionally called a theorem. https://erich-friedman.github.io/packing/ https://erich-friedman.github.io/packing/ is probably one of the best known on HN.
Jtsummers 5y ago
Or as noted in the article: > But one of the distinctive things about math is that our papers aren’t just records of experiments we did elsewhere. In experimental sciences, the experiment is the “real work” and the paper is just a description of it. But in math, the paper, itself, is the “real work”. And > And that means that you can replicate a math paper by reading it.

the_watcher 5y ago

The social science and medical replication crisis seems like it would be far more impactful than a mathematics crisis, right? Politicians, policy-makers, doctors, etc. all make decisions based on potentially flawed or outright incorrect studies in a way that I don't think is true for the equivalents in math, simply because there aren't decisions and policies up for debate related to much of them (if I am wrong about this, please correct me).

smartscience 5y ago
If a flawed mathematical paper were used as the basis for what then became a flawed cryptography algorithm, I can see that having impact if the bad guys noticed the flaw first. But yes, I expect examples like that would be comparatively rare.
- the_watcher 5y ago
  Thanks for this, cryptography is a good example where this could be a problem.
- not2b 5y ago
  In cryptography the math is almost always the strongest part, and it is the side-channel attacks and implementation mistakes that let the bad guys in. When it is the math, the flaw is often that the algorithm has all the desirable properties proved in a number of papers, but has some exploitable structure that analysts can turn into an attack.
IshKebab 5y ago
I think politicians generally don't make policy decisions based on science. I live in the UK and an obvious example is drug policy. They even fired the scientist whose drug research they didn't like. But if the science agrees with a decision they've already made then they're happy to use if for justification, even if the science is junk (e.g. the crazy fines on taking children out of school in the UK).

gus_massa 5y ago

> When I’ve served as a peer reviewer I’ve read the papers closely and checked all the steps of the proofs, and that means that I have replicated the results. The side effect is that math papers have an insane long time to publication. Perhaps 6 months, or 1 year or more if you are unlucky. In physics, the publication time is like 3 month. Something like 1 month for the first review and then two months for making small changes suggested by the referee and discussing with the editor. As a side^2 effect, some citation index of the journals count only the citations during the first year. But the papers that have the citation are sleeping over the reviewer desk during that year, so the number is lower than the real number.

Barrin92 5y ago

why not turn the premise from the article around. Instead of suggesting that math might also have a replication crisis, why not question whether the whole replication crisis thing was overblown and effectively more of an ideological attack on a few disciplines. Errors in science, disagreement and lack of reproducibility I think are common and prevalent but it doesn't necessarily imply that a discipline as a whole doesn't make progress. The obsession with statistical accuracy and 'science as bookkeeping' mentality seems fairly new to begin with, and science did just fine before we even had the means to verify every single thing ever published. It kind if ignores the dynamic nature of science. Most of what is published probably has close to zero impact regardless of whether its right or wrong, but paradigm changing research generally asserts itself. Science is evolutionary in that sense, it's full of mistakes but stumbles towards correct solutions at uneven tempo. In a sense you can just look at it like VC investment. Nine times out of ten individual things don't work, but the sector overall works, the market economy is full of grifters and failed businesses, but it doesn't matter that much. So, maybe half of math is bullshit but so is everything else but in math people just say "whatever" until they find something good, whereas in psychology people use it as an opportunity to hack away at it.

burnished 5y ago
This isn't turning the problem around in the same sense. In order to effectively turn a problem around you need to use its complement, and then it should be a binary proposition. Your example where you suppose an idealogical attack fails here because it is not the only other explanation, you haven't turned the problem around in the same way that turning "what is the probability this action had an effect" around can become "what is the probability that this action had no effect".
quanticle 5y ago
It kind if ignores the dynamic nature of science. Most of what is published probably has close to zero impact regardless of whether its right or wrong, but paradigm changing research generally asserts itself. The problem is that psychology, unlike physics, doesn't really have a paradigm. There's the old saying, "Extraordinary claims require extraordinary evidence." But for that heuristic to be effective, you need a standard for what counts as extraordinary. Extraordinary compared to what? In phsyics, there are two well-established paradigms (relativity and quantum mechanics), which establish what counts as ordinary, and what counts as extraordinary. So, for example, if you're making a claim that the distribution of dark matter in the cosmos more clumpy than predicted by existing models, of that the energy level of a particular field is 12 MeV rather than 10, those are ordinary claims, which can be accomodated by tweaks to the existing paradigm. But if you're saying that the speed of light has varied over the history of the universe, or that all subatomic particles are actually tiny vibrating string-like structures, well, that's going to require a lot more evidence. In psychology, it's much more difficult to have that kind of intuition. Take the concept of priming, for example. Is claiming that people walk more slowly when they're encouraged to think of things that make them feel old extraordinary? It makes a certain sort of intuitive sense, but, on the other hand, there's absolutely no causal mechanism suggested. So when a number of priming studies fail spectacularly under replication [1], I don't know what to think. I don't have a good sense for how much of psychology is overturned by the replication failure, in the same sense that I'd have for physics if it turned out that e.g. the speed of light is a variable rather than a constant. [1]: https://mindhacks.com/2017/02/16/how-replicable-are-the-social-priming-studies-in-thinking-fast-and-slow/ https://mindhacks.com/2017/02/16/how-replicable-are-the-soci...

svat 5y ago

Despite the title (a title with a question in it invites people to comment without reading the post, even more than the usual already high level), this is a really good post IMO, with valuable insights into not just mathematics but also the replication crisis elsewhere. (And it does discuss Mochizuki's claimed proof of the abc conjecture, and links to MathOverflow on formalization with proof assistants, to a recent paper discussing Vladimir Voevodsky's views, etc.) This from the first part is sound: > The replication crisis is partly the discovery that many major social science results do not replicate. But it’s also the discovery that we hadn’t been trying to replicate them, and we really should have been. In the social sciences we fooled ourselves into thinking our foundation was stronger than it was, by never testing it. But in math we couldn’t avoid testing it. But the post doesn't stop there! The second part of the post (effect sizes etc), with the examples of power posing and the impossibly hungry judges, is even more illuminating. Thanks!

zitterbewegung 5y ago

Math doesn’t have a replication crisis at all it has a comprehension crisis. Since proofs are programs one can basically say that mathematical theorems are incredibly detailed software that is completely open source and invites people to identify programs that don’t work and or fix issues. A famous one is Fermats last theorem which needed a fix but was largely right. Others have said that it takes 6 months to a year to get published. The other thing with math is the fact that you can get completely scooped and your work is worthless. Edit: I am using "proofs are programs" very loosely and yes Theorems are much more than programs as other commenters have pointed out.

contravariant 5y ago
A more notorious example of the comprehension crisis would be Mochizuki's claimed proof of the abc conjecture. So far fairly few people are willing to claim they both understand and agree with the several hundred pages of 'proof'.
- exdsq 5y ago
  https://en.wikipedia.org/wiki/Shinichi_Mochizuki https://en.wikipedia.org/wiki/Shinichi_Mochizuki It's a fun rabbit hole to go down :)
- zitterbewegung 5y ago
  I was tempted to use that but Fermat's last theorem is known to the general public for much longer and has a resolution.
  - contravariant 5y ago
    It was a good example, and definitely one of the more important examples of how finding and fixing mistakes in mathematics is supposed to happen. I figured the abc-conjecture would provide a nice contrasting example.
Quekid5 5y ago
An interesting perspective on programs-as-proof is I-forget-his-name-but... the mathematician who made really bold claims, if only you'd study under his tutelage for a number of years to understand this whole new terminology he invented. With programs-as-proof it really wouldn't matter. It's either "computer says yes" or "compu'er says noooo". EDIT: Whoop, sibling post mentioned, it's Mochizuki.

Simplicitas 5y ago

Steven Pinker takes a decent crack at "Statistical Significance" in his new book Rationality (https://en.wikipedia.org/wiki/Rationality_(book) https://en.wikipedia.org/wiki/Rationality_(book)), which underpins a lot of this and is mentioned in this piece. And I'm still grappling with this part of the book. lol

mjw1007 5y ago

I think it's a good article. But it's worth considering the possibility that mathematics could have fallen, and could still fall, into a state where false results are frequently published, and isn't 'protected' by anything special in the nature of the field or its practitioners. Just as you might find yourself asking "why did city A fall into the grip of organised crime, and not city B?". You might look for answers in the methods of police recruitment or a strong history of respect for the rule of law or anything like that, but it might turn out that the answer is really just "city A got unlucky".

raphlinus 5y ago

This is a thoughtful and thought-provoking blog post. I think it's worth asking similar questions of computer science. I think you'll find some math-like patterns -- there's basically no chance Quicksort or any other fundamental algorithm paper will fail to replicate -- and some patterns which will fail to replicate, like in software engineering. Some of the early results on pseudorandom generators and hash functions aren't holding up well, but I think that's just progress. We understand the problem a whole lot better than we did back then. Perhaps more interesting is the literature on memory models. The original publications of the Java and C11 memory models had lots of flaws, which took many years to fix (and that process might not be totally done). I worry that there are a bunch of published results that are similarly flawed but just haven't gotten as much scrutiny.

ahelwer 5y ago
In competitive programming you could basically assume the pseudocode in a paper is not literally correct and requires some tweaking to work, despite a “proof” of its correctness. Particularly with string algorithms.
- sterlind 5y ago
  long time no see! there's a couple levels there: rote translating pseudocode into your target language isn't likely to pan out well. so instead you run the pseudocode in your mind, develop an intuition on how it works, and that's the "replication" bit this post talks about with reviewing math papers. but both the pseudocode and your code will likely have edge cases you didn't handle. this isn't a problem for math - that's the category of common trivial/easily fixable proof errors that don't really affect the paper. but they're a problem for machines that run them literally. maybe a good compromise strategy for formal verification is to declare the insight of the algorithm - recurrence relation or whatever - as an axiom, and then use the prover to whack the tricky edge cases.
Analemma_ 5y ago
The parts of CS that are the most math-like (which include fundamental algorithms) don't have a replication crisis, but the ones that are the most social-science like probably do, or would. I would bet large sums of money that a lot of the literature on stuff like "does OOP lead to better code", "does IDE syntax highlighting lead to fewer bugs" etc. would fail to replicate if anyone bothered trying. The thing is, the general sense I get is that people in CS already have so little confidence in these results that it's not even considered worth the time to try and refute them. Which doesn't exactly speak well of the field!

schuyler2d 5y ago

You could also argue that Math had its replication crisis in the 17th-19th centuries. E.g. infinite series "proofs" that were eventually shown to be flawed methodologies. This and other crises led to grounding modern mathematics with set theory, Zermelo–Fraenkel axioms, etc and understanding what's possible (e.g. Godel's theorem). Psychology and other social sciences are barely a century old.

fshbbdssbbgdd 5y ago
A more recent example is the Italian school of algebraic geometry, where it was discovered in the mid 20th century that many claimed proofs were faulty (and some “proven” results were incorrect).
zmgsabst 5y ago
Mathematics is having a replication crisis and people pay so little attention they don’t know. That replication crisis has led to efforts in formal verification such as HoTT, Lean, etc. https://homotopytypetheory.org/ https://homotopytypetheory.org/ https://xenaproject.wordpress.com/2021/06/05/half-a-year-of-the-liquid-tensor-experiment-amazing-developments/ https://xenaproject.wordpress.com/2021/06/05/half-a-year-of-...
- Victerius 5y ago
  How is math experiencing a replication crisis? > This site serves to collect and disseminate research, resources, and tools for the investigation of homotopy type theory, and hosts a blog for those involved in its study. > Exactly half a year ago I wrote the Liquid Tensor Experiment blog post, challenging the formalization of a difficult foundational theorem from my Analytic Geometry lecture notes on joint work with Dustin Clausen. ???
- dwohnitmok 5y ago
  No by and large mathematics is not having a replication crisis. As the blog post in your link states: > Question: Was the proof in [Analytic] found to be correct? > Answer: Yes, up to some usual slight imprecisions. This has been the case for almost all math formalization efforts. Even when (very rarely) proofs were revealed to be incorrect, the result was salvageable.

dilawar 5y ago

A bit cynical take is that it is harder to mislead in theory? Your mistakes will get caught and you risk losing your reputation so it less likely that someone will publish a proof without double triple checking. Theory subject has faster verificarion loops. Verification and review is harder in experimental works and it might take few decades before someone finds an error, or even collect resources to verify the claim. So maybe it's harder to fight 'publish or perish' deamons in experimental sciences?

verisimi 5y ago

"We get away with it becuase we can be right for the wrong reasons—we mostly only try to prove things that are basically true." Apart from the replication crisis, the other crisis that is not really talked about is funding. Academic funding basically comes from one of 3 (connected) sources - government, corporations and the military. Somehow or other - these 3 sources have pretty much the same or non-conflicting aims. These aims relate to power and control. This is actually the largest crisis, IMO. Given that information, we can re-assess the quoted statement the author makes. Perhaps its not that they are proving things that are "basically true". Its that right or wrong do not matter. What does matter is that the answers provided meet the agenda of those funding the study. The answer is not that important as long as it is supportive of whatever agenda is in play. I believe this is the case for the replication crisis in science also. A replication "crisis" is only a crisis if you are attempting to achieve truth and greater understanding. But truth and understanding are only ostensible reasons, not the actual ones. What these studies are actually doing is creating a parallel construction - the aim is actually for studies to appear 'truthey', without actually being so. What studies should actual do is increase the funder's power, wealth extraction abilities, etc. If you doubt this and think that truth matters, consider this. Surely we should have cracked the best diet for people by now? But there is no common understanding of what is good or bad to eat - if anything there is more confusion. The reason of course, is that there's no money in recommending whole foods or whatever. However, there is money in drugs to make people 'better'. And money in making diet so confusing that people eat themselves into trouble. Anyway, if you are in the business of governing or monetising the masses, truth and understanding is the last thing you want. Far better to have a story that gives you control, or extracts money. Such is life under fascist governance (where fascist = corporation + governance working together).

scoofy 5y ago

The paper here seems to make absolutely zero distinction between deductive and inductive reasoning, which should be the entire point. Math is an arbitrary framework, built from arbitrary axioms. It is deductive. Thus, all proofs are simply deduction. The knowledge here is positive knowledge, we can show things are true, false, or undecidable. There may be errors, but those are errors in execution. Psychology is not built on a framework. It is inductive, we are trying to find axioms that map to the data we collect. Thus, all papers are trying to add/build it's arbitrary framework. The only knowledge here is negative knowledge, falsification, we know only what is a failed hypothesis. There will be errors, both in execution, and there will also be statistical errors in experimental results. The entire point of the replication crisis is that we don't publish or pay attention to results that are boring, so the framework we build is built on skewed data. We don't reject previously popular papers that are now unfalsifiable (the idea that the now unfalsifiable Milgram experiment is still taught in every university psychology dept should be outrageous). The boring results need to be weight statistically against the interesting results, but aren't, etc. Nobody out there is arguing whether or not the axiom of choice is a true axiom? It sort of doesn't matter, it can't matter, because it's arbitrary by definition. You can't have a replication crisis inside of a deductive framework without changing the framework. This doesn't happen too often, but we did see this during the shift from neutonian to einsteinian physics. The study of philosophy of science is fairly obscure, but is the center of this discussion.

netizen-936824 5y ago
I don't believe I'm familiar with an Ingram experiment, any chance you could expand on this? I was also unable to find something with a quick internet search, however Milgram was suggested to me.
- scoofy 5y ago
  Milgram experiment, sorry... must have got hit by autocorrect there: https://en.wikipedia.org/wiki/Milgram_experiment https://en.wikipedia.org/wiki/Milgram_experiment You can add the Stanford Prison Experiment (Zimbardo) to the list of, now unfalsifiable, experiments that inexplicably are regularly taught in university settings when they cannot possibly provide any useful data to the sciences: https://en.wikipedia.org/wiki/Stanford_prison_experiment https://en.wikipedia.org/wiki/Stanford_prison_experiment The idea that many if not most universities create educational programs based on these very obvious problematic studies, leads me to believe that the scientific community can be as guilty of info-tainment bias as the general public. I only wish that I'd been able to continue in academia so that i could have at least a small voice in changing this problematic dynamic we live with. They literally made a movie about the Zimbardo experiment in 2015. It couldn't be more obvious that people care more that it be true than whether it might be false: https://en.wikipedia.org/wiki/The_Stanford_Prison_Experiment_(film) https://en.wikipedia.org/wiki/The_Stanford_Prison_Experiment...

curt15 5y ago

It did: https://en.wikipedia.org/wiki/Italian_school_of_algebraic_geometry https://en.wikipedia.org/wiki/Italian_school_of_algebraic_ge...

ocschwar 5y ago

There is one corner of math that does have a replication crisis. Just as we compare programming languages by how "ergonomic" they are to learn and use, mathematicians do come up with novel notation systems to try to improve the ergonomic state of their field, and since "ergonomics" is another way to say "esthetics", and is proved or disproved by user testing, that is where replication gets hard. The inventor of category theory's wiring diagrams, for example, has claimed that he could get middle schoolers to understand them. I suspect that success has not been replicated.

mlyle 5y ago
> The inventor of category theory's wiring diagrams, for example, has claimed that he could get middle schoolers to understand them. I suspect that success has not been replicated. Maybe I should try. I like category theory and I am a middle school teacher and like teaching electives on improbable things :D

lacker 5y ago

The root of the replication crisis in social sciences is not just that many papers fail to replicate, but that there is no way to clearly resolve a result that fails to replicate. A paper claims that pre-K increases test scores a decade later, another paper claims it doesn't, and there's no clear resolution. The disagreement just festers, with both sides citing research that supports their opinion. The argument often "spills out" into the public sphere. In mathematics and computer science, there are many errors in published papers. However, once you point out an error, it's usually pretty straightforward to resolve whether it's really an error or not. Often there is a small error which can be fixed in a small way. Exceptions like the abc conjecture are rare.

umvi 5y ago

My completely subjective opinion is that at the highest levels of math, there are only a handful of people that are even capable of peer reviewing, and their time is in high demand. Wiles's proof of Fermat's Last Theorem is like 120 pages long and he first delivered it disguised as a class to a bunch of grad students who barely understood any of it and hence gave no feedback. Because this is Fermat's Last Theorem which is famous, eventually people in the math community that understood Wiles's work reviewed it and found an error. Had it been a 120 page proof of some not famous problem like random chessboard thought experiments, it probably could go years without anyone seriously looking at it.

amelius 5y ago

Because math is not science? https://philosophy.stackexchange.com/questions/14818/is-mathematics-considered-a-science https://philosophy.stackexchange.com/questions/14818/is-math...

kkoncevicius 5y ago
I wanted to post the same thing, so was looking for a top comment addressing it and glad to have found it. It's weird how many people, even on HN, do not get this distinction. Math gets thrown in with all the sciences but it itself is not a science - it doesn't have experiments, it doesn't follow the scientific method. Mathematics cannot have a replication crisis because there is nothing to replicate. In math correctness is tested not by redoing something but by double-checking the original work for mistakes.

unemphysbro 5y ago

2+2=4

jll29 5y ago

Mathematics is fully deductive (so it is not strictly speaking a science - it is not empirical, but it's own kind of discipline, like logics, if you don't consider that part of mathematics), so replication can be attempted by machine, and as the authors wrote proof-checkers find more and more mistakes. A water-tight proof essentially is the trace or log showing a series of mechanical deductive steps. But because journals publish human proofs, one of the following cases can arise: - A proof can be wrong and the underlying conjecture can still be true (i.e. a theorem). - A proof can be wrong and the conjecture can be false - these kinds of errors need to be corrected with utmost urgency, because they lead to follow-up downstream errors. - A proof can be right and recognized as such by peers, in which case it gets published and everybody is happy. - A proof can also be right and be contested by peers. This is when proofs have gaps that are "obvious" to some, but not believed by others. Because a proof is a sequence of steps that are either previously derived or self-evident, and people simply differ what they consider self-evident (as an aside, in school I was criticized in maths for "skipping steps" and in English for "jumping thoughts"; to me it seemed obvious where things were going but the teachers obviously needed "comments to the code", so I learned to insert baby steps, and everyone was happy). Because of the last case, it is important to get down to the most formal, fine-grained, nitpicky, atomic level, so everyone can agree that each micro-step is self-evident, and thankfully we can dedicate this to machines nowadays (provers like the "Isabelle" system - https://www.cl.cam.ac.uk/research/hvg/Isabelle/ https://www.cl.cam.ac.uk/research/hvg/Isabelle/), at least to an extent. Ironically, that's not how real proofs in mathematics journals look like at all. They're written in prose, and often written in a surprisingly "meta" style (we had algebra lectures where the professor talked about "colored roosters" and how they behave, which is the name of some structure in the algebraic sub-field of Ramsey theory).

graderjs 5y ago

Because it's largely not an experimental science, but rather is groups of people convincing themselves they know what someone is talking about. As the article pretty much says. The article also suggests an "exceptionalism" of maths in that their results hold up even when their methods don't. In other words math papers are full of "unimportant flaws and errors" but even though that's the case when corrected or verified, somehow the conclusions of the papers mostly hold up, in the author's opinion. But I tend to think this is more because, math is groups of people (and formal verification systems) convincing themselves they know what someone else is talking about, so confirmation bias is probably in effect. The limitations of a self-consistent set of axioms after all... You may say that formal verification systems are perfect oracles... But if papers have "unimportant flaws and errors" it's possible that those systems have them, too, right? It's also possible that as those systems were written by people, they reflect and somehow seek to confirm the biases of those people or their understandings (or misunderstandings) about mathematical knowledge. So I don't think maths is quite as exceptional as this author hopes it might. Tho Maybe it is

xiphias2 5y ago

There are lots of cryptographical protocols with mathematical proofs that contain errors that break the security of the protocols themselves. A version MuSig in Bitcoin was a protocol like that, though luckily the mistake was caught by the cryptographic community when the authors wanted to publish the proof at a conference.

topaz0 5y ago

It seems that a lot of commenters here have not read the post, and are simply posting their own opinions about how to answer the question in the title. If that is you, and you are interested in the question, I recommend reading it in full -- there are a lot of interesting points there.

chrisbrandow 5y ago

when I was getting my PhD in chemistry, the replication crisis had not been recognized (widely anyway). as it has become more widely discussed I've vacillated between annoyed that people don't make the distinction between the hard/simplef sciences and the fields that study much more complicated, multi-casual topics and being self critical that perhaps chemistry also suffers from a widespread problem. I still don't know the answer but I suspect there is something to the simple/complicated distinction. I do recall that there was this one professor that couldn't replicate his seminal work that basically got him tenure and it was all very dramatic. the kind of story that we would share with each other as a byword. Aslo, I wonder if the errors get caught more quickly because in chemistry and physics to a degree, most progress is built directly on previous results. so you can't have bad results just floating unnoticed for very long.

mateo1 5y ago
The thing is, mistakes will always be made and corrected. Maybe the experiment worked the first time because someone smoked in the room yesterday and a trace amount of contaminant made it in the solution and catalysed a reaction. Maybe your supplier had a minor contamination on your reactants. The reasons are endless. The issues I have with the "replication crisis" are two: malicious publishing of false results for personal or organisational profit, and malicious interpretation of prestigious publications as the one and only immutable truth. Both of which are social and political problems. Scientific publishing was functioning fine so far for the same reason the old internet was full of true information while it didn't have to: there was little to be gained by lying. Now that gov policies are build on phd papers, things are changing.

pmjones 5y ago

"All science is either physics, or stamp collecting." (Ernest Rutherford)

aaron695 5y ago

mathteddybear 5y ago

One time when I was still a sophomore a friendly professor shared a story how one of his colleagues had a math paper accepted, and found an error in the proof only after it was published. So he had another paper in that journal next, with the refutation of his earlier work. And that feat was worth double the point score awarded to the institute by the scientific powers that be. I'm sure the excuse is that errors in math papers these days don't have the same kind of impact as mistakes in, let's say, medicine. Later on my guess was that this will change when we will have a editor software that is easy to use, wysiwyg, renders math texts as well as latex, and actually the edited document is integrated with the formal proof, beneath the fancy elegant phrases and formulas. Then I quit a job and enrolled for phd studies but that's another story.

dragonwriter 5y ago

Because math isn't science, so there is nothing to replicate. If the analysis is sound, that's it, there's no question of whether the physical results the analysis was applied to are fraudulent, measurement error, due to improper setup, etc. Why is this even a question? Analytical and empirical fields face fundamentally dissimilar challenges.

throwawayarnty 5y ago

The replication crisis is not largely due to the math/stats used in studies. It comes from trying to make inference from noisy and often confounded experiments from limited data.

robertlagrant 5y ago

The article's linked article on social priming has a fun typo: 4.10 The “Lacy Macbeth Effect”

_xnmw 5y ago

Unrelated, but the replication crisis in medicine is the primary reason why anti-vaxxers have a right to be skeptical, even if they're wrong and spout nonsense. Now we *know* that a large percentage of cited medical research is fraught with uncertainty and is potentially non-replicable. Therefore, people have a right to be skeptical of "the science" even if they don't have a coherent counterargument. It is extremely reasonable to be suspicious of statistical methodologies in vaccine safety studies and adverse reaction reporting; there are statistical errors and even outright fraud all over the scientific landscape, so why not here? And given the extremely perverse financial and political incentives found in this particular pandemic environment, I would be extremely surprised if there was NO foul play happening, because you would have to be a hero to ignore these incentives.

bongoman37 5y ago

srcreigh 5y ago

My summary: in social science, measurements are coarse which means the statistics cannot evaluate subtle changes. In math, logic is very precise so subtle changes can be measured. The author makes a point that both social science people and mathematicians are trying to prove subtle things which are actually probably true. In other words the social science replication crisis is because the experiments are often impossible to perform consistently when the effect is subtle, leading to the use of inconsistent lucky draws to demonstrate things.

deleted 5y ago

[deleted]

JPLeRouzic 5y ago

I like this sentence from the link: > Many papers have errors, yes—but our major results generally hold up, even when the intermediate steps are wrong!

croes 5y ago

In mathematics there is no measurement uncertainty

qwerty456127 5y ago

How can there be errors in math papers if math formulae are trivial to check formally using a computer?

dagw 5y ago
Because the current state of the art of automated proof checking is still trying to check all the proofs you'll see in a undergraduate program and nowhere near able to check state of the art proofs.
asdf_snar 5y ago
Math formulae are not trivial to check formally using a computer. Mathematical software is in its infancy relative to the sophistication of mathematical objects defined in the literature.

benreesman 5y ago

Not being a formally trained mathematician or psychologist but being someone who’s seen plenty of organization/institutional mechanism design: is it possible that e.g.psychology research might become more mathematically rigorous if there were more incentives for serious mathematics people to study psychology? I appreciate at first glance that might seem a bit vacuous: but there’s a ton of money and way better policy at stake if we understood behavioral economics better, and that’s at least an adjacent field to psychology right?

IvanK_net 5y ago

Just a fun fact: there are dozens of papers released each year, which claim that they have proven that P=NP, or P is not equal NP. E.g. they "prove" it with soap bubbles: https://arxiv.org/pdf/cs/0406056.pdf https://arxiv.org/pdf/cs/0406056.pdf

pfortuny 5y ago
Well to be honest, things on arxiv are not properly speaking “papers” in the academic sense. At least in Maths.

nemo44x 5y ago

In Soviet Russia, mathematicians, physicists, etc came to the same conclusions as their enemies in the USA. However, the social science results, although unified in each country, were almost always different and in conflict. Social sciences are politics and nothing more. It’s an opinion of how the world ought to be.

ContrarianBrit 5y ago

Very interesting! As a social scientist, the replication crisis gets me seriously down, making me doubt the integrity of my field. It's good to see that there's not such a problem in maths...

LatteLazy 5y ago

Replication crisis come down to funding. If you publish bad results in maths, you lose funding and get fired and humiliated. If you publish bad (but exciting or popular) results in social sciences, you get more funding and acclaim. All subjects are basically (sadly) products. Maths, Physics and Engineering are products that have to (mostly, eventually) actually work. Politics, Sociology, Gender Studies etc are more about entertainment. Som fields (eg nutrition) are a hybred of the two: some serious factual research, some fads and fashion.

phkahler 5y ago

Because math doesnt rely on experiments.

deleted 5y ago

[deleted]

maliker 5y ago

Maybe there isn’t a math replication crisis, but there is a kind of math fragmentation crisis. Sub-fields of mathematics have gotten so deep that specialists can’t communicate with one another [0]. I’ve heard this expressed by other mathematicians as having only 2 or 3 other people in the world that they can discuss their work with. The level of solitary study necessary to make progress is another sign of this phenomenon. Andrew Wiles famously spent six years alone in his attic to come up with his proofs for Fermat’s Last Theorem. [0] https://sites.math.rutgers.edu/~zeilberg/Opinion104.html https://sites.math.rutgers.edu/~zeilberg/Opinion104.html