4 ms·

I believe this rule of thumb will come to fail. The combination of superhuman mathematical reasoning and synthesis in upcoming AI models plus the rapid build-ou

by nilkn 2mo ago

I believe this rule of thumb will come to fail. The combination of superhuman mathematical reasoning and synthesis in upcoming AI models plus the rapid build-out of scalable formal verification infrastructure means this exponential in math is going to take off quite explosively, and we've barely seen anything yet. Mathematics is going to decisively move beyond human ability fairly soon (within our lifetimes, if not much more abruptly). It seems abundantly clear to me that much of the work will only be immediately accessible to AI, and rather than trying to explain all of it back to humans we will rather focus on explaining the portions that humans would benefit disproportionately from understanding.

skybrian 2mo ago
Maybe that will be true when it's math with practical applications, but most theoretical math isn't like that. If it's not practical and it's not for mathematians to understand, what good is it?
- esafak 2mo ago
  One day it might be for the AI's pleasure, the same way it has heretofore been for ours. Or if you prefer, as a byproduct of its programming to acquire knowledge.
- cma 2mo ago
  You could have one really hard to understand proof of a theorem and then a lot of interesting human-understandable stuff that relies on that theorem. We already have lots of proofs with oracles, where you can work out consequences of what kind of structures and solutions could exist if you had some magic thing to solve a hard part, so it just seems like a variation on that. Many people learn calculus or even the real numbers without understanding the complete formalization from set theory.
- nilkn 2mo ago
  We have thousands of years of precedent that suggests that breakthroughs in mathematics tend to accumulate into broader technology breakthroughs in other domains. Why does this tend to be the case, even when some of the smartest people in the world have historically predicted incorrectly that certain branches of math would forever be useless (e.g., number theory)? I can only offer my own theory on that, but my guess is that mathematics is simply a predictive framework based on pattern compression. A more powerful pattern compression framework accelerates every single field that relies on pattern recognition or prediction of the unknown based on patterns.
  - skybrian 2mo ago
    It sounds like the idea is to turn on a math generator and keep running it until it generates something interesting. And it might be fun to try it. But if it’s too much output to read and we don’t understand the output either, how does anyone recognize when it’s done something that’s practically interesting? The output might make a cool screen saver as-is, but we probably need a way to evaluate it somehow.
    - nilkn 2mo ago
      Yes, of course we'd need a way to evaluate it. I don't right now have a fully conceived answer to what that will look like. But I'm confident at least in saying we would not evaluate it, like Tao is suggesting, by only accepting something once a human can easily teach it unassisted to another human. That sets the bar dramatically too low and would quickly become an extraordinary impediment to progress. You'd have to think of yourself less like a researcher and more like the director of the world's largest research institute. It's highly unlikely you'll understand or even care about every single paper every one of your researchers is producing, but you'll care about the overall research direction and whether the intermediate results are accumulating into outcomes you consider meaningful. How to do this where the institute is based on superhuman AI mathematicians is an unsolved problem, but I see no reason to imagine it's unsolvable. Let me make up an example of where I could imagine this going. Something we essentially cannot do right now is predict coarse-grained phenomena from systems that involve millions or trillions or more of interacting components. Over hundreds/thousands of years of experiment and theory we've derived laws that essentially do this in a few special cases, but we have no systematic theoretical way of doing it in general, and frankly I think it's beyond human ability. Whatever deep patterns or structures exist for doing this in a general way I think are simply out of reach for us.
    - energy123 2mo ago
      This is relevant, but only after AI has solved all the open problems including Millenium problems. Until then, as AI keeps solving harder open problems, people will pay attention and be interested.
  - chongli 2mo ago
fultonn 2mo ago
People say similar things about automation of software engineering. Different, but similar. I'm deeply suspicious. I do not yet have a concise statement for why, but a lot of literature on the sociology of knowledge work sort of points at my thoughts. Section 5 of the Thurston article cited by Tao touches the elephant. Raduchel's article on the economics of software [2] also touches it. I've tried to put words to this for a few years. I think I'm just going to start writing versions of it as see if that helps me shape the thought into something more concise. So, in the spirit of this article's style, here are some postulates: 1. There is a sociological process happening in the production function during knowledge work. 2. That production function and the associated sociological process spans years or even decades, and must outlast many of the artifacts that are produced during the early years of the function. 3. You cannot get the right lines of code or the right theorems proved without running that sociological process alongside the artifact production process. 4. It is impossible to completely separate the sociological process from the artifact construction process. If you just iterate on artifacts then too much of the required hidden state is lost to make progress in the right direction. This is true even if you include distilled artifacts capturing pieces of the sociological process (eg meeting notes, documentation, commit logs, prompts). 5. So you need that sociological process, or something like it, to still happen. 6. For a lot of knowledge work that process plays out in extremely high-fidelity social interactions [3] that we have not yet captured in the datasets that would be required to reproduce those dynamics. 7. And even if we do collect that data, our current architectures and training algorithms and hardware would be useless given the size of the datasets. So: the technology today gives us the ability to iterate on the production of artifacts. But it does not sufficiently simulate the social process which gives rise to the Right artifacts. This isn't exactly what I actually think, but it's a version of the thing that I intuit when I watch heavy use of AI in both software projects and formalization projects. And simulating that process feels way harder than people are currently assuming. [1] https://arxiv.org/pdf/math/9404236 https://arxiv.org/pdf/math/9404236 Section 5. [2] https://www.nationalacademies.org/read/11587/chapter/11 https://www.nationalacademies.org/read/11587/chapter/11 pp 166-168. [3] there is a reason we still gather in-person around white boards, and why doing so is more crucial for some types of work than others.
- energy123 2mo ago
deleted 2mo ago
[deleted]
addag 2mo ago
[dead]

by nilkn 2mo ago

skybrian 2mo ago

Maybe that will be true when it's math with practical applications, but most theoretical math isn't like that. If it's not practical and it's not for mathematians to understand, what good is it?

esafak 2mo ago
One day it might be for the AI's pleasure, the same way it has heretofore been for ours. Or if you prefer, as a byproduct of its programming to acquire knowledge.
cma 2mo ago
You could have one really hard to understand proof of a theorem and then a lot of interesting human-understandable stuff that relies on that theorem. We already have lots of proofs with oracles, where you can work out consequences of what kind of structures and solutions could exist if you had some magic thing to solve a hard part, so it just seems like a variation on that. Many people learn calculus or even the real numbers without understanding the complete formalization from set theory.
nilkn 2mo ago
We have thousands of years of precedent that suggests that breakthroughs in mathematics tend to accumulate into broader technology breakthroughs in other domains. Why does this tend to be the case, even when some of the smartest people in the world have historically predicted incorrectly that certain branches of math would forever be useless (e.g., number theory)? I can only offer my own theory on that, but my guess is that mathematics is simply a predictive framework based on pattern compression. A more powerful pattern compression framework accelerates every single field that relies on pattern recognition or prediction of the unknown based on patterns.
- skybrian 2mo ago
  It sounds like the idea is to turn on a math generator and keep running it until it generates something interesting. And it might be fun to try it. But if it’s too much output to read and we don’t understand the output either, how does anyone recognize when it’s done something that’s practically interesting? The output might make a cool screen saver as-is, but we probably need a way to evaluate it somehow.
  - nilkn 2mo ago
    Yes, of course we'd need a way to evaluate it. I don't right now have a fully conceived answer to what that will look like. But I'm confident at least in saying we would not evaluate it, like Tao is suggesting, by only accepting something once a human can easily teach it unassisted to another human. That sets the bar dramatically too low and would quickly become an extraordinary impediment to progress. You'd have to think of yourself less like a researcher and more like the director of the world's largest research institute. It's highly unlikely you'll understand or even care about every single paper every one of your researchers is producing, but you'll care about the overall research direction and whether the intermediate results are accumulating into outcomes you consider meaningful. How to do this where the institute is based on superhuman AI mathematicians is an unsolved problem, but I see no reason to imagine it's unsolvable. Let me make up an example of where I could imagine this going. Something we essentially cannot do right now is predict coarse-grained phenomena from systems that involve millions or trillions or more of interacting components. Over hundreds/thousands of years of experiment and theory we've derived laws that essentially do this in a few special cases, but we have no systematic theoretical way of doing it in general, and frankly I think it's beyond human ability. Whatever deep patterns or structures exist for doing this in a general way I think are simply out of reach for us.
  - energy123 2mo ago
    This is relevant, but only after AI has solved all the open problems including Millenium problems. Until then, as AI keeps solving harder open problems, people will pay attention and be interested.
- chongli 2mo ago

fultonn 2mo ago

People say similar things about automation of software engineering. Different, but similar. I'm deeply suspicious. I do not yet have a concise statement for why, but a lot of literature on the sociology of knowledge work sort of points at my thoughts. Section 5 of the Thurston article cited by Tao touches the elephant. Raduchel's article on the economics of software [2] also touches it. I've tried to put words to this for a few years. I think I'm just going to start writing versions of it as see if that helps me shape the thought into something more concise. So, in the spirit of this article's style, here are some postulates: 1. There is a sociological process happening in the production function during knowledge work. 2. That production function and the associated sociological process spans years or even decades, and must outlast many of the artifacts that are produced during the early years of the function. 3. You cannot get the right lines of code or the right theorems proved without running that sociological process alongside the artifact production process. 4. It is impossible to completely separate the sociological process from the artifact construction process. If you just iterate on artifacts then too much of the required hidden state is lost to make progress in the right direction. This is true even if you include distilled artifacts capturing pieces of the sociological process (eg meeting notes, documentation, commit logs, prompts). 5. So you need that sociological process, or something like it, to still happen. 6. For a lot of knowledge work that process plays out in extremely high-fidelity social interactions [3] that we have not yet captured in the datasets that would be required to reproduce those dynamics. 7. And even if we do collect that data, our current architectures and training algorithms and hardware would be useless given the size of the datasets. So: the technology today gives us the ability to iterate on the production of artifacts. But it does not sufficiently simulate the social process which gives rise to the Right artifacts. This isn't exactly what I actually think, but it's a version of the thing that I intuit when I watch heavy use of AI in both software projects and formalization projects. And simulating that process feels way harder than people are currently assuming. [1] https://arxiv.org/pdf/math/9404236 https://arxiv.org/pdf/math/9404236 Section 5. [2] https://www.nationalacademies.org/read/11587/chapter/11 https://www.nationalacademies.org/read/11587/chapter/11 pp 166-168. [3] there is a reason we still gather in-person around white boards, and why doing so is more crucial for some types of work than others.

energy123 2mo ago

deleted 2mo ago

[deleted]

addag 2mo ago

[dead]