5 ms·

> Every day, it grows harder and harder to contain a mental map of recent relevant progress by simple virtue of the amount being produced. And by opening the d

> Every day, it grows harder and harder to contain a mental map of recent relevant progress by simple virtue of the amount being produced. And by opening the door to LLM-generated results, you'll see greater and greater amounts without any hope of ever navigating this field again without machine help. It's a little like a software project which more and more gets extended by a AI agents with less and less review by human software engineers and in the end the complexity and spaghetti design are so incomprehensible by humans that the maintenance requires an AI agent. The risk is that math as a whole (the field itself) will experience that effect.

biztos 5mo ago
I'm no mathematician but it seems like if this happens, we get to a quite intriguing place as a species. Say we achieve interstellar travel, but nobody actually knows how it works. Or we cure cancer, but the "cure" requires a microrobotic implant, and it runs as a blackbox AI, and only the other AIs can make one, and there's no guarantee they will know how to make one tomorrow. Or we solve global warming but it requires giant cooling machines running 24/7 and again, nobody knows how it works, but with the added bonus that the planet is cooked if they ever stop working.
- f055 5mo ago
  So I guess sci-fi movies were right all along. Nobody in Star Wars knows how hyperspace travel works, it just works. The little robots know everything but almost no human bothers to care. People just carry on with their bickering lives while the bots whiz in the background, and these robots are astonished at human inefficiency every single time, but rarely do anything about it. And people are still people.
  - cyclopeanutopia 5mo ago
    That's only because movies like Star Wars are not sci-fi movies, but more like westerns in space.
    - wartywhoa23 5mo ago
      Sounds of explosions and engine whine in vacuum beg to differ!
- tsunamifury 5mo ago
  “Few-shun drive. Mmmm. Yes.” https://youtu.be/pfNS2kWf5cY?si=SH6_QC0bCspV-ngz https://youtu.be/pfNS2kWf5cY?si=SH6_QC0bCspV-ngz There are comments that truly reveal a future horrifying and true. Few of them. But I count yours among them. But I’d argue also that airplanes already achieve this complexity to some degree as well as microprocessors.
  - darkwater 5mo ago
    > as well as microprocessors I mean, microprocessors have been on the "impossible to bootstrap from scratch in a short period of time" for 20 years already.
    - fragmede 5mo ago
      More like 50. The Intel 4004 came out in 1971, and there's no way to bootstrap even that level of technology after some sort of doomsday scenario.
      - darkwater 5mo ago
        But I would say that processor like the Z80 are so simple and well-known and documented in the wild that they would have some chance to be bootstrapped again with what would realistically survive, one way or another, in a doomsday scenario. But yeah 20 year is too short, it's more like 40 years.
- suncemoje 5mo ago
  I think anything is and will be explainable. Like in the OpenAI proof, I’m sure they were able to understand the solution 100% and could even drill down and ask more clarifying questions to the model. After all, the point of science is so that knowledge can be made logically transparent. If something can’t be explained, it isn’t really understood yet — and the same applies to model outputs. The only question is how much effort it takes to surface the explanation.
  - worldsayshi 5mo ago
    And more intelligence should give an opportunity to increase explain-ability rather than just complexity. It can potentially explain the proof at the level of the listener. Make visualizations. Etc.
  - fauigerzigerk 5mo ago
    I think explanation is itself a rather complex concept. At what point do we consider something as explained? Usually it has to do with identifying some causal factors and their relationships so that we can intervene and explore counterfactuals. But in many cases we are forced to act on the basis of incomplete explanations (e.g. in medicine). I think there will be regulation that requires some users of AI to provide an explanation upon request. For instance, banks could be required to "explain" why you didn't get that loan. What if the decision is based on a credit score that includes some AI prediction that ultimately relies on the entire training corpus? The bank can give you a list of factors that play into the decision but they may not be able to explain deterministically why a very similar customer did get that loan. At that point I think we're going to resort to statistics that prove a lack of bias against certain protected characteristics, but that's not really an explanation, is it? I think we will never get useful and complete explanations for everything that AI does. Society will just accept some explanation-like thing or proxy and move on.
  - latexr 5mo ago
    > I’m sure they were able to understand the solution 100% and could even drill down and ask more clarifying questions to the model. If they understood it 100%, what clarification is needed?
  - 5mo ago
- SyzygyRhythm 5mo ago
  That's already how civilization works. There's no one person that knows everything about (say) modern food production, from top to bottom. If it ever stopped working (because too much knowledge was lost somehow), most people would die. And yet the system seems fairly resilient. Mostly, only local knowledge ever seems to be necessary to keep the whole thing running. Super-intelligence (or even just super-normal-intelligence) might expand the scope of what constitutes local knowledge but it will still run into limits somewhere.
  - lefra 5mo ago
    > There's no one person that knows everything about (say) modern food production True, but it is possible to assemble a team of people that does, with backup for each person. There's also teachers and written knowledge to educate new team members. That's what makes it resilient. I think that's a very different situation from what's decribed.
    - pishpash 5mo ago
      People still do grow their own food for self sufficiency. I am sure there will be luddites who live in self-sufficient communes like the Amish.
      - timschmidt 5mo ago
        People lump them together because of an anti-technology reputation, but I don't think most Amish would have trucked with Luddites. Amish tend to avoid actively participating in popular social movements, and oppose violence and property destruction.
    - fc417fc802 5mo ago
      Agreed, the food production analogy doesn't really work because the issue is the scale of the problem. On the one end there's the realm where you need a few specialists and a small group could potentially figure the entire thing out from scratch given a bit of time and effort. And then at the opposite extreme there's the realm where everything is built on a giant pyramid of artifacts that currently work, just keeping each individual piece running day to day requires a dedicated expert, and the combined stack took hundreds or thousands of lifetime equivalents to develop. The idea being that once a toolchain becomes sufficiently complex if you ever have to bootstrap it again for whatever reason you won't be able to speedrun the process the way you might naively expect. I think modern chip production likely already reached this point several decades ago. As evidence I'll point out that China only recently achieved EUV and remains several nodes behind despite directing an obscene amount of resources towards the initiative.
- worldsayshi 5mo ago
  Why can't we (or AI) invent ways to explain information that makes it much more digestible? And the solutions simpler? Why is it necessary to continue to increase complexity when we get better intelligence? Can't we find more simple solutions? Or at least more explainable.
  - pishpash 5mo ago
    Is particle physics digestible even if it is explainable? Some things are not simple, cannot be not abstract, and will not be understood by most, or all, people.
    - Windchaser 5mo ago
      Or biochemistry. It’s complex, and there’s nothing we can do about it.
- ajjl1011 5mo ago
  Which, funnily enough, is exactly what this anthropic person is saying - https://www.theguardian.com/technology/2026/may/21/ai-nobel-prize-winning-discovery-robots-jack-clark-anthropic https://www.theguardian.com/technology/2026/may/21/ai-nobel-... See comment about "scientific equipment that people hadn’t conceived of but which worked"
- wartywhoa23 5mo ago
  Noble "save the humankind using the tech nobody fucking understands" textbook goals like curing cancer, solving global warming and achieving interstellar travel would always turn up when owners of trillions of dollars place orders on positive AI narratives, but in reality all of that will wither down to "It's what plants crave! It's got electrolytes."
  - ocimbote 5mo ago
    That is fascinating how the more knowledge and reasoning we can get our hands on and actually produce, the higher the risk of us, as a species, to become actually much dumber. It's hard to describe the feeling of seeing intelligence being delegated increasingly to AI. If that's not a pivotal moment, a revolution, I don't know what is.
    - wartywhoa23 5mo ago
      If you're not familiar with it, I recommend looking up the Taoist concept of overdevelopment. Sums it all up perfectly.
    - withinboredom 5mo ago
      > That is fascinating how the more knowledge and reasoning we can get our hands on and actually produce, the higher the risk of us, as a species, to become actually much dumber. This has always been true. There was a time where someone had to teach farming to others and that information had to spread and be passed down. Eventually, farmers became better than hunter-gatherers and they became known as hunters. The information on what was safe to gather for civilisation got passed down as 'safe to eat on the hunt' because the farmers were farming. The civilisation collectively "forgets" foraged foods as that knowledge becomes niche. Does that mean we got dumber?
  - ffaccount2 5mo ago
    >ve AI narratives, but in reality all of that WILL wither down to... Looks like you're pretty sure of that. Every time I see argument like this delivered with confidence I wonder how is it different from, say, digital calculators. Or better yet, books - Greek philosophers moaned that young people will stop understanding anything and just check books when they want to know anything.
- Rebuff5007 5mo ago
  "All models are wrong, but some are useful" What your describing is already how a lot of science, technology, and engineering works!
- ivell 5mo ago
  We don't know how many things in nature work. For example, we don't fully understand our own brain. As long as it can be replicated, we are fine. In case of AI we have a better chance to understand what it is doing through chain of thought and explainability. Nature never gave us that..
- latexr 5mo ago
  > Or we solve global warming but it requires giant cooling machines running 24/7 That’s not “solving it”, that’s putting a bandaid on it. Solving it would mean correcting the underlying issue to the point it’s no longer a problem which requires maintenance. Managing symptoms is not curing the disease.
- loandbehold 5mo ago
  This is pretty much what underlies AI doomer argument of people like EY. Humans will gradually hand over civilization to black box AI they can't understand. As AI becomes more complex and powerful it will be harder and harder to control.
  - pishpash 5mo ago
    That assumes everyone will do so. Some people won't, and it's not clear you need a large number of such, a priesthood if you like, to survive as a species without AI.
    - ben_w 5mo ago
      There's a lot of un-contacted (or voluntarily self-isolating, it's hard to tell which) tribes around the world; things go poorly for them when businesses with power tools decide they want the resources of the land they live in, even when they're theoretically protected by law of the nation whose borders both the tribes and the businesses both find themselves within.
- nashadelic 5mo ago
  I've been thinking about this and I believe the best place to be is a scientist who keeps looking at an AI's output, prods it in the right directions, verifies the proofs, fixes and fills gaps, takes the proof to production with safety, risks etc mitigated and then distribution with a company wrapped around the discovery. I think it wouldn't be black-boxed as much and will require a lot more understanding and reviewing to trust and productize it.
- ben_w 5mo ago
  The first two are open enough that they may be as you say*, but we already know how to solve global warming, it's more of how much do we want to. Green energy and transport technology is now at the point where people save the world and get rich trying, just as fast as they can build the factories. Food's climate impact is harder, because the problem isn't technical, it's convincing people to give up beef (and other things, but mostly beef). * quantum mechanics and general relativity are famously difficult to get to grips with
- juvoly 5mo ago
  Like the discovery of penicillin by Alexander Fleming in 1928?
- bayindirh 5mo ago
  Welcome to the "Hyperion Cantos". The book doesn't deviate from what you have envision, or the future you envision doesn't deviate from the book, I may say.
- mxkopy 5mo ago
  I doubt we’d build anything IRL without understanding how it works first. And we’re pretty good at putting 2+2 together once we have the pieces, for a lot of these things we don’t even have those. After all AI can just explain it to us atp
- wombatpm 5mo ago
  So knowledge becomes meta stable. There was an AIDS drug in the 90’s that we stopped being able to make. IIRC Apparently there were two different crystal/folding structures for the compound and the desired one was not the lowest energy. After years of production, the wrong version was produced and they could no longer make the correct version do to contamination. And every facility that tried to study it, wound up no longer being able to make the correct version. It was like a real life ice-IX situation. Scary that changing weights or model parameters could lead to the same thing happening with knowledge.
  - cromulent 5mo ago
    Fascinating. I had to read up on this. Apparently it was Ritonavir and polymorphism. https://en.wikipedia.org/wiki/Ritonavir https://en.wikipedia.org/wiki/Ritonavir
- UltraSane 5mo ago
  This is like in Hyperion where AIs invented the farcaster wormhole portals and no human or team of experts ever came close to understanding how they worked.
Certhas 5mo ago
The amount of papers produced passed the point of being digestible by humans a long time ago. I do think we will need to find a way to get away from publishing papers. But I thought that before the AI came along and made mediocre papers something you can produce in a day. The academic system seems utterly incapable of self-correcting on this point though. We haven't even managed to get rid of for-profit publishers. So how this all will go down is anybodies guess right now.
CosmicHazard 5mo ago
and the longer that this goes on, then the decision making and critical thinking will be offloaded to the LLM's as well.

biztos 5mo ago

I'm no mathematician but it seems like if this happens, we get to a quite intriguing place as a species. Say we achieve interstellar travel, but nobody actually knows how it works. Or we cure cancer, but the "cure" requires a microrobotic implant, and it runs as a blackbox AI, and only the other AIs can make one, and there's no guarantee they will know how to make one tomorrow. Or we solve global warming but it requires giant cooling machines running 24/7 and again, nobody knows how it works, but with the added bonus that the planet is cooked if they ever stop working.

f055 5mo ago
So I guess sci-fi movies were right all along. Nobody in Star Wars knows how hyperspace travel works, it just works. The little robots know everything but almost no human bothers to care. People just carry on with their bickering lives while the bots whiz in the background, and these robots are astonished at human inefficiency every single time, but rarely do anything about it. And people are still people.
- cyclopeanutopia 5mo ago
  That's only because movies like Star Wars are not sci-fi movies, but more like westerns in space.
  - wartywhoa23 5mo ago
    Sounds of explosions and engine whine in vacuum beg to differ!
tsunamifury 5mo ago
“Few-shun drive. Mmmm. Yes.” https://youtu.be/pfNS2kWf5cY?si=SH6_QC0bCspV-ngz https://youtu.be/pfNS2kWf5cY?si=SH6_QC0bCspV-ngz There are comments that truly reveal a future horrifying and true. Few of them. But I count yours among them. But I’d argue also that airplanes already achieve this complexity to some degree as well as microprocessors.
- darkwater 5mo ago
  > as well as microprocessors I mean, microprocessors have been on the "impossible to bootstrap from scratch in a short period of time" for 20 years already.
  - fragmede 5mo ago
    More like 50. The Intel 4004 came out in 1971, and there's no way to bootstrap even that level of technology after some sort of doomsday scenario.
    - darkwater 5mo ago
      But I would say that processor like the Z80 are so simple and well-known and documented in the wild that they would have some chance to be bootstrapped again with what would realistically survive, one way or another, in a doomsday scenario. But yeah 20 year is too short, it's more like 40 years.
suncemoje 5mo ago
I think anything is and will be explainable. Like in the OpenAI proof, I’m sure they were able to understand the solution 100% and could even drill down and ask more clarifying questions to the model. After all, the point of science is so that knowledge can be made logically transparent. If something can’t be explained, it isn’t really understood yet — and the same applies to model outputs. The only question is how much effort it takes to surface the explanation.
- worldsayshi 5mo ago
  And more intelligence should give an opportunity to increase explain-ability rather than just complexity. It can potentially explain the proof at the level of the listener. Make visualizations. Etc.
- fauigerzigerk 5mo ago
  I think explanation is itself a rather complex concept. At what point do we consider something as explained? Usually it has to do with identifying some causal factors and their relationships so that we can intervene and explore counterfactuals. But in many cases we are forced to act on the basis of incomplete explanations (e.g. in medicine). I think there will be regulation that requires some users of AI to provide an explanation upon request. For instance, banks could be required to "explain" why you didn't get that loan. What if the decision is based on a credit score that includes some AI prediction that ultimately relies on the entire training corpus? The bank can give you a list of factors that play into the decision but they may not be able to explain deterministically why a very similar customer did get that loan. At that point I think we're going to resort to statistics that prove a lack of bias against certain protected characteristics, but that's not really an explanation, is it? I think we will never get useful and complete explanations for everything that AI does. Society will just accept some explanation-like thing or proxy and move on.
- latexr 5mo ago
  > I’m sure they were able to understand the solution 100% and could even drill down and ask more clarifying questions to the model. If they understood it 100%, what clarification is needed?
- 5mo ago
SyzygyRhythm 5mo ago
That's already how civilization works. There's no one person that knows everything about (say) modern food production, from top to bottom. If it ever stopped working (because too much knowledge was lost somehow), most people would die. And yet the system seems fairly resilient. Mostly, only local knowledge ever seems to be necessary to keep the whole thing running. Super-intelligence (or even just super-normal-intelligence) might expand the scope of what constitutes local knowledge but it will still run into limits somewhere.
- lefra 5mo ago
  > There's no one person that knows everything about (say) modern food production True, but it is possible to assemble a team of people that does, with backup for each person. There's also teachers and written knowledge to educate new team members. That's what makes it resilient. I think that's a very different situation from what's decribed.
  - pishpash 5mo ago
    People still do grow their own food for self sufficiency. I am sure there will be luddites who live in self-sufficient communes like the Amish.
    - timschmidt 5mo ago
      People lump them together because of an anti-technology reputation, but I don't think most Amish would have trucked with Luddites. Amish tend to avoid actively participating in popular social movements, and oppose violence and property destruction.
  - fc417fc802 5mo ago
    Agreed, the food production analogy doesn't really work because the issue is the scale of the problem. On the one end there's the realm where you need a few specialists and a small group could potentially figure the entire thing out from scratch given a bit of time and effort. And then at the opposite extreme there's the realm where everything is built on a giant pyramid of artifacts that currently work, just keeping each individual piece running day to day requires a dedicated expert, and the combined stack took hundreds or thousands of lifetime equivalents to develop. The idea being that once a toolchain becomes sufficiently complex if you ever have to bootstrap it again for whatever reason you won't be able to speedrun the process the way you might naively expect. I think modern chip production likely already reached this point several decades ago. As evidence I'll point out that China only recently achieved EUV and remains several nodes behind despite directing an obscene amount of resources towards the initiative.
worldsayshi 5mo ago
Why can't we (or AI) invent ways to explain information that makes it much more digestible? And the solutions simpler? Why is it necessary to continue to increase complexity when we get better intelligence? Can't we find more simple solutions? Or at least more explainable.
- pishpash 5mo ago
  Is particle physics digestible even if it is explainable? Some things are not simple, cannot be not abstract, and will not be understood by most, or all, people.
  - Windchaser 5mo ago
    Or biochemistry. It’s complex, and there’s nothing we can do about it.
ajjl1011 5mo ago
Which, funnily enough, is exactly what this anthropic person is saying - https://www.theguardian.com/technology/2026/may/21/ai-nobel-prize-winning-discovery-robots-jack-clark-anthropic https://www.theguardian.com/technology/2026/may/21/ai-nobel-... See comment about "scientific equipment that people hadn’t conceived of but which worked"
wartywhoa23 5mo ago
Noble "save the humankind using the tech nobody fucking understands" textbook goals like curing cancer, solving global warming and achieving interstellar travel would always turn up when owners of trillions of dollars place orders on positive AI narratives, but in reality all of that will wither down to "It's what plants crave! It's got electrolytes."
- ocimbote 5mo ago
  That is fascinating how the more knowledge and reasoning we can get our hands on and actually produce, the higher the risk of us, as a species, to become actually much dumber. It's hard to describe the feeling of seeing intelligence being delegated increasingly to AI. If that's not a pivotal moment, a revolution, I don't know what is.
  - wartywhoa23 5mo ago
    If you're not familiar with it, I recommend looking up the Taoist concept of overdevelopment. Sums it all up perfectly.
  - withinboredom 5mo ago
    > That is fascinating how the more knowledge and reasoning we can get our hands on and actually produce, the higher the risk of us, as a species, to become actually much dumber. This has always been true. There was a time where someone had to teach farming to others and that information had to spread and be passed down. Eventually, farmers became better than hunter-gatherers and they became known as hunters. The information on what was safe to gather for civilisation got passed down as 'safe to eat on the hunt' because the farmers were farming. The civilisation collectively "forgets" foraged foods as that knowledge becomes niche. Does that mean we got dumber?
- ffaccount2 5mo ago
  >ve AI narratives, but in reality all of that WILL wither down to... Looks like you're pretty sure of that. Every time I see argument like this delivered with confidence I wonder how is it different from, say, digital calculators. Or better yet, books - Greek philosophers moaned that young people will stop understanding anything and just check books when they want to know anything.
Rebuff5007 5mo ago
"All models are wrong, but some are useful" What your describing is already how a lot of science, technology, and engineering works!
ivell 5mo ago
We don't know how many things in nature work. For example, we don't fully understand our own brain. As long as it can be replicated, we are fine. In case of AI we have a better chance to understand what it is doing through chain of thought and explainability. Nature never gave us that..
latexr 5mo ago
> Or we solve global warming but it requires giant cooling machines running 24/7 That’s not “solving it”, that’s putting a bandaid on it. Solving it would mean correcting the underlying issue to the point it’s no longer a problem which requires maintenance. Managing symptoms is not curing the disease.
loandbehold 5mo ago
This is pretty much what underlies AI doomer argument of people like EY. Humans will gradually hand over civilization to black box AI they can't understand. As AI becomes more complex and powerful it will be harder and harder to control.
- pishpash 5mo ago
  That assumes everyone will do so. Some people won't, and it's not clear you need a large number of such, a priesthood if you like, to survive as a species without AI.
  - ben_w 5mo ago
    There's a lot of un-contacted (or voluntarily self-isolating, it's hard to tell which) tribes around the world; things go poorly for them when businesses with power tools decide they want the resources of the land they live in, even when they're theoretically protected by law of the nation whose borders both the tribes and the businesses both find themselves within.
nashadelic 5mo ago
I've been thinking about this and I believe the best place to be is a scientist who keeps looking at an AI's output, prods it in the right directions, verifies the proofs, fixes and fills gaps, takes the proof to production with safety, risks etc mitigated and then distribution with a company wrapped around the discovery. I think it wouldn't be black-boxed as much and will require a lot more understanding and reviewing to trust and productize it.
ben_w 5mo ago
The first two are open enough that they may be as you say*, but we already know how to solve global warming, it's more of how much do we want to. Green energy and transport technology is now at the point where people save the world and get rich trying, just as fast as they can build the factories. Food's climate impact is harder, because the problem isn't technical, it's convincing people to give up beef (and other things, but mostly beef). * quantum mechanics and general relativity are famously difficult to get to grips with
juvoly 5mo ago
Like the discovery of penicillin by Alexander Fleming in 1928?
bayindirh 5mo ago
Welcome to the "Hyperion Cantos". The book doesn't deviate from what you have envision, or the future you envision doesn't deviate from the book, I may say.
mxkopy 5mo ago
I doubt we’d build anything IRL without understanding how it works first. And we’re pretty good at putting 2+2 together once we have the pieces, for a lot of these things we don’t even have those. After all AI can just explain it to us atp
wombatpm 5mo ago
So knowledge becomes meta stable. There was an AIDS drug in the 90’s that we stopped being able to make. IIRC Apparently there were two different crystal/folding structures for the compound and the desired one was not the lowest energy. After years of production, the wrong version was produced and they could no longer make the correct version do to contamination. And every facility that tried to study it, wound up no longer being able to make the correct version. It was like a real life ice-IX situation. Scary that changing weights or model parameters could lead to the same thing happening with knowledge.
- cromulent 5mo ago
  Fascinating. I had to read up on this. Apparently it was Ritonavir and polymorphism. https://en.wikipedia.org/wiki/Ritonavir https://en.wikipedia.org/wiki/Ritonavir
UltraSane 5mo ago
This is like in Hyperion where AIs invented the farcaster wormhole portals and no human or team of experts ever came close to understanding how they worked.

Certhas 5mo ago

The amount of papers produced passed the point of being digestible by humans a long time ago. I do think we will need to find a way to get away from publishing papers. But I thought that before the AI came along and made mediocre papers something you can produce in a day. The academic system seems utterly incapable of self-correcting on this point though. We haven't even managed to get rid of for-profit publishers. So how this all will go down is anybodies guess right now.

CosmicHazard 5mo ago

and the longer that this goes on, then the decision making and critical thinking will be offloaded to the LLM's as well.