2 ms·

You seem determined not to understand this, but to spell it out even further: The point is not that the companies are deliberately trying to harm maths but that

You seem determined not to understand this, but to spell it out even further: The point is not that the companies are deliberately trying to harm maths but that in pursuing their own goals, which are different to those of mathematicians, they are doing so. Their goals and their methods of achieving them are not aligned with those of mathematicians. This is, incidentally, exactly what alignment means in AI as well: a paperclip maximiser harms humans not because it is intentionally trying to do so but as a side effect of trying to achieve its own objectives, which are not aligned with those of humans (which include not having their entire environment turned into paperclips). The word is being used entirely correctly, and I'm sure the nod to the AI usage is quite deliberate.

zozbot234 14d ago
> Their goals and their methods of achieving them are not aligned with those of mathematicians. This is exactly the assumption that Tao is smuggling in with "misalignment" talk and then refusing to elaborate on any further. Is the issue that AI companies are willfully refusing to provide mathematical insight that they could provide (because they have diverging underlying "goals" to those of human mathematicians) or are they merely working under a capability gap, where current AIs can awkwardly settle major open questions but are not smart enough to provide the kind of understanding and insight that the community of human mathematicians relies on? These are two very different problems and by foregrounding the word "misalignment" in his letter so openly (as opposed to talking about AI capability to provide valued insight), Tao is picking the more adversarial reading with zero proof or motivation.
- omnicognate 14d ago
  His view is the "capability gap" one, and you can just read the the statement and Tao's other writings for far more eloquent explanations of the difference between a raw LLM proof dump and real mathematical insight, and what it takes to get from the former to the latter, than I can provide. Again, nobody is accusing anyone of deliberately trying to harm mathematics. It's about misaligned objectives. Eg. Tao wrote the following before the Navier-Stokes announcement (referring to exactly the project OpenAI was undertaking): > At this point, I would not be surprised if one could batter out such an extension by pouring an enormous amount of compute and AI assistance at such a task. But such an exercise does not particularly hold my interest; I am far more interested in digesting the proof methods and extracting out the key new insights uncovered by this approach. And then went further to say that such activity could be actively harmful to the field. (All this can be read on Mathstodon: https://mathstodon.xyz/@tao https://mathstodon.xyz/@tao). The misalignment is that this activity that he and 24 other Fields medalists think is actively harmful to their field is deemed by OpenAI and others to be worth ploughing vast financial, human and compute resources into.
  - zozbot234 14d ago
    > His view is the "capability gap" one That's the far more sensible reading, so thanks for confirming I guess. But then the misalignment talk is pretty clearly a distraction. > ...And then went further to say that such activity could be actively harmful to the field. If true (and there is as of yet insufficient evidence of this), that's merely a contingent fact about very real institutional misalignment within the human mathematical community, not about AI itself or even AI frontier labs. There's simply zero inherent reason why providing a bare truth value or a completely inscrutable proof about the status of some open conjecture should make that entire subfield of math "contaminated" for the foreseeable future when it comes to extracting further human-relevant insight. That's the misalignment we should be caring about.
    - SpicyLemonZest 14d ago
      The declaration is about that institutional misalignment. They want to change the norms of the mathematical community, so that producing inscrutable proofs is a low-value activity nobody cares much about rather than a high-value activity that AI frontier labs can make headlines by performing. Their ask of the frontier labs is to please be aware of the problems with the old norms and not exploit them during the transition period; they agree this doesn't have much to do with AI itself, which they acknowledge is a powerful technology that will improve and accelerate mathematical research.
      - zozbot234 14d ago
        Broadly agreed, with a key proviso: producing inscrutable proofs has negligible value as a mathematician's finished output but that doesn't make it a "low-value activity" in and of itself. Ultimately, the status of these proof-like objects as a raw input into mathematical practice will probably be comparable to any other sort of computationally-driven https://en.wikipedia.org/wiki/Experimental_mathematics https://en.wikipedia.org/wiki/Experimental_mathematics . These are not new problems: "computer" used to be a job description for humans before it was the name of a machine, but we now view raw computations as a trivial matter that's not worthy of any human credit.
- freehorse 14d ago
  It is really not complicated at all. And if tao's post is a bit vague, the letter signed by many mathematicians is imo very clear. AI can be used to advance/deepen understanding, or it can be used to superficially go settle a whole bunch of open problems in a field without helping really in understanding them. It all depends on who uses the AI and why. Essentially, it is exactly the same concept as using the AI as a course tutor vs having it do your homework. Or using the AI to write millions of lines of code that nobody can actually read, vs keeping overview of what is going on. In math it is probably worse because there is no objective function to maximise. Some people here think that the objective function of mathematics is to prove things, which is actually wrong. Mathematicians are not only maximising an objective function, they are also defining the objective function they need to maximise (they are defining which problems to study). The problem with AI/ML is that it can be pretty good when the goal is to maximise a set objective function, but not to set intentions and goals themselves. I would not call that a "capability gap" because we can actually get to have very useful and smart AI systems without ever reaching that point.
- deleted 14d ago
  [deleted]
psychoslave 14d ago
Even mathematical community as a whole is not that aligned with the stated goals. Not to blame mathematicians in general, its just a general anthropological platitude that the group actual behavior is not fully matching what people from the group want to pretend even in full honesty. Though it doesn't take even an outsider to question the matching here, just look what the only guy who settled a millennium prize what he thought about the community. Or ask some actual PhD, postdoc, or even better someone who dropped along the way, how does it feel to go through this community. Not sure the goals exposed in this article are really helped with the community as it is. One again, this doesn't mean everywhere encompasses the same issues and everyone is acting badly on is own. But if we want to accept happy pink shiny depiction of a community, we should be giving the same generosity to other communities à priori. All the more as AI and mathematical community do share a large set of common individuals.

zozbot234 14d ago

> Their goals and their methods of achieving them are not aligned with those of mathematicians. This is exactly the assumption that Tao is smuggling in with "misalignment" talk and then refusing to elaborate on any further. Is the issue that AI companies are willfully refusing to provide mathematical insight that they could provide (because they have diverging underlying "goals" to those of human mathematicians) or are they merely working under a capability gap, where current AIs can awkwardly settle major open questions but are not smart enough to provide the kind of understanding and insight that the community of human mathematicians relies on? These are two very different problems and by foregrounding the word "misalignment" in his letter so openly (as opposed to talking about AI capability to provide valued insight), Tao is picking the more adversarial reading with zero proof or motivation.

omnicognate 14d ago
His view is the "capability gap" one, and you can just read the the statement and Tao's other writings for far more eloquent explanations of the difference between a raw LLM proof dump and real mathematical insight, and what it takes to get from the former to the latter, than I can provide. Again, nobody is accusing anyone of deliberately trying to harm mathematics. It's about misaligned objectives. Eg. Tao wrote the following before the Navier-Stokes announcement (referring to exactly the project OpenAI was undertaking): > At this point, I would not be surprised if one could batter out such an extension by pouring an enormous amount of compute and AI assistance at such a task. But such an exercise does not particularly hold my interest; I am far more interested in digesting the proof methods and extracting out the key new insights uncovered by this approach. And then went further to say that such activity could be actively harmful to the field. (All this can be read on Mathstodon: https://mathstodon.xyz/@tao https://mathstodon.xyz/@tao). The misalignment is that this activity that he and 24 other Fields medalists think is actively harmful to their field is deemed by OpenAI and others to be worth ploughing vast financial, human and compute resources into.
- zozbot234 14d ago
  > His view is the "capability gap" one That's the far more sensible reading, so thanks for confirming I guess. But then the misalignment talk is pretty clearly a distraction. > ...And then went further to say that such activity could be actively harmful to the field. If true (and there is as of yet insufficient evidence of this), that's merely a contingent fact about very real institutional misalignment within the human mathematical community, not about AI itself or even AI frontier labs. There's simply zero inherent reason why providing a bare truth value or a completely inscrutable proof about the status of some open conjecture should make that entire subfield of math "contaminated" for the foreseeable future when it comes to extracting further human-relevant insight. That's the misalignment we should be caring about.
  - SpicyLemonZest 14d ago
    The declaration is about that institutional misalignment. They want to change the norms of the mathematical community, so that producing inscrutable proofs is a low-value activity nobody cares much about rather than a high-value activity that AI frontier labs can make headlines by performing. Their ask of the frontier labs is to please be aware of the problems with the old norms and not exploit them during the transition period; they agree this doesn't have much to do with AI itself, which they acknowledge is a powerful technology that will improve and accelerate mathematical research.
    - zozbot234 14d ago
      Broadly agreed, with a key proviso: producing inscrutable proofs has negligible value as a mathematician's finished output but that doesn't make it a "low-value activity" in and of itself. Ultimately, the status of these proof-like objects as a raw input into mathematical practice will probably be comparable to any other sort of computationally-driven https://en.wikipedia.org/wiki/Experimental_mathematics https://en.wikipedia.org/wiki/Experimental_mathematics . These are not new problems: "computer" used to be a job description for humans before it was the name of a machine, but we now view raw computations as a trivial matter that's not worthy of any human credit.
freehorse 14d ago
It is really not complicated at all. And if tao's post is a bit vague, the letter signed by many mathematicians is imo very clear. AI can be used to advance/deepen understanding, or it can be used to superficially go settle a whole bunch of open problems in a field without helping really in understanding them. It all depends on who uses the AI and why. Essentially, it is exactly the same concept as using the AI as a course tutor vs having it do your homework. Or using the AI to write millions of lines of code that nobody can actually read, vs keeping overview of what is going on. In math it is probably worse because there is no objective function to maximise. Some people here think that the objective function of mathematics is to prove things, which is actually wrong. Mathematicians are not only maximising an objective function, they are also defining the objective function they need to maximise (they are defining which problems to study). The problem with AI/ML is that it can be pretty good when the goal is to maximise a set objective function, but not to set intentions and goals themselves. I would not call that a "capability gap" because we can actually get to have very useful and smart AI systems without ever reaching that point.
deleted 14d ago
[deleted]

psychoslave 14d ago

Even mathematical community as a whole is not that aligned with the stated goals. Not to blame mathematicians in general, its just a general anthropological platitude that the group actual behavior is not fully matching what people from the group want to pretend even in full honesty. Though it doesn't take even an outsider to question the matching here, just look what the only guy who settled a millennium prize what he thought about the community. Or ask some actual PhD, postdoc, or even better someone who dropped along the way, how does it feel to go through this community. Not sure the goals exposed in this article are really helped with the community as it is. One again, this doesn't mean everywhere encompasses the same issues and everyone is acting badly on is own. But if we want to accept happy pink shiny depiction of a community, we should be giving the same generosity to other communities à priori. All the more as AI and mathematical community do share a large set of common individuals.