5 ms·

If AGI is ever achieved, it would open the door to recursive self improvement that would presumably rapidly exceed human capability across any and all fields, i

If AGI is ever achieved, it would open the door to recursive self improvement that would presumably rapidly exceed human capability across any and all fields, including AI development. So the AI would be improving itself while simultaneously also making revolutionary breakthroughs in essentially all fields. And, for at least a while, it would also presumably be doing so at an exponentially increasing rate. But I think we're not even on the path to creating AGI. We're creating software that replicate and remix human knowledge at a fixed point in time. And so it's a fixed target that you can't really exceed, which would itself already entail diminishing returns. Pair this with the fact that it's based on neural networks which also invariably reach a point of sharply diminishing returns in essentially every field they're used in, and you have something that looks much closer to what we're doing right now - where all competitors will eventually converge on something largely indistinguishable from each other, in terms of ability.

thinkingtoilet 1y ago
>And, for at least a while, it would also presumably be doing so at an exponentially increasing rate. Why would you presume this? I think part of a lot of people's AI skepticism is talk like this. You have no idea. Full stop. Why wouldn't progress be linear? As new breakthroughs come, newer ones will be harder to come by. Perhaps it's exponential. Perhaps it's linear. No one knows.
- robwwilliams 1y ago
  Or bottlenecked by data availability just like we humans are. Nothing will be exponential if a loop in the real world of science and engineering is involved.
  - jatora 1y ago
    Coding and algorithmic advance does not require real world experimentation.
    - ceh123 1y ago
      Right but for self improving AI, training new models does have a real world bottleneck: energy and hardware. (Even if the data bottleneck is solved too)
  - MarcelOlsz 1y ago
    Aren't we bottlenecked by not having any "prior art", as in not having reverse engineered any thinking machine like even a fly's brain? We can't even agree on a definition of consciousness and still don't understand the brain or how it works (to the extent that reverse engineering it can tell us something).
- jatora 1y ago
  It's a logical presumption. Researchers discover things. AGI is a researcher that can be scaled, research faster, and requires no downtime. Full stop if you dont find that obvious you should probably figure out where your bias is coming from. Coding and algorithmic advance does not require real world experimentation.
  - throwaway48476 1y ago
    This is fundamentally flawed. There are upper bounds of efficiency that are laws of nature. To assume AI would be supernatural is magical thinking.
    - solumunus 1y ago
      And yet it seems to be the prevailing opinion even among very smart people. The “singularity” it’s just presumed. I’m highly skeptical to say the least. Look how much energy it’s taking to engineer these models which are still nowhere near AGI. When we get to AGI it won’t be immediately super intelligent and perhaps it never will be. Diminishing returns surely apply to anything that is energy based?
    - me-vs-cat 1y ago
      Natural intelligence appears supernatural from our current understanding, so it's not surprising that AGI also appears so.
      - goatlover 1y ago
        Neither appears supernatural from a scientific understanding.
  - thinkingtoilet 1y ago
    Prove it.
  - dkersten 1y ago
    If AGI has human capability, why would we think it could research any faster than a human? Sure, you can scale it, but if an LLM takes, say, $1 million a year to run an AGI instance, but it costs only $500k for one human researcher, then it still doesn’t get you anywhere faster than humans do. It might scale up, it might not, we don’t know. We won’t know until we reach it. We also don’t know if it scales linearly. Or if it’s learning capability and capacity will able to support exponential capability increase. Our current LLM’s don’t even have the capability of self improvement or learning even if they were capable: they can accumulate additional knowledge through the context window, but the models are static unless you fine tune or retrain them. What if our current models were ready for AGI but these limitations are stopping it? How would we ever know? Maybe it will be able to self improve but it will I’ll take exponentially larger amounts of training data. Or exponentially larger amounts of energy. Or maybe it can become “smarter” but at the cost of being larger to the point where the laws of physics mean it has to think slower, 2x the thinking but 2x the time, could happen! What if an AGI doesn’t want to improve? Far too many unknowns to say what will happen.
- hennell 1y ago
  No one knows, but it's a reasonable assumption surely. If you're theorising a AGI, that has recursive self improvement, exponential improvements seem almost unavoidable. AGI improves understanding of electronics, physics etc, that improves the AGI leading to new understandings and so on. Add in that new discoveries in one field might inspire the AGI/humans to find things in others and it seems hard to imagine a situation where theres not a lot of progress everywhere (at least theoretical progress, building new things might be slower / more costly then reasoning they would work.) Where I'm skeptical of AI would be in the idea an LLM can ever get to AGI level, if AGI is even really possible, and if the whole thing is actually viable. I'm also very skeptical that the discoveries of any AGI would be shared in ways that would allow exponential growth; licenses stopping using their AGI to make your own, copyright on the new laws of physics and royalties on any discovery you make from using those new laws etc.
  - thinkingtoilet 1y ago
    >If you're theorising a AGI, that has recursive self improvement, exponential improvements seem almost unavoidable. Prove it. Also, AI will need resources. Hardware. Water. Electricity. Can those resources be supplied at an exponential rate? People need to calm down and stop stating things as truth when they literally have no idea.
    - johnisgood 1y ago
      Which is why I am an agnostic. :)
    - mangodrunk 1y ago
      Well said. It does seem that many who speculate on this are not taking into account limits that more/faster processing won’t actually help much. Say an algorithm is proven to be O(n!) for all cases, at a certain size of n, there’s not much that can be done if the algorithm is needed as is.
  - deleted 1y ago
    [deleted]
- frognumber 1y ago
  I always consider different options when planning for the future, but I'll give the argument for exponential: Progress has been exponential in the generic. We made approximately the same progress in the past 100 years as the prior 1000 as the prior 30,000, as the prior million, and so on, all the way back to multicellular life evolving over 2 billion years or so. There's a question of the exponent, though. Living through that exponential growth circa 50AD felt at best linear, if not flat.
  - FabHK 1y ago
    So you concede that there's nothing special about AI versus earlier innovations?
  - ileonichwiesz 1y ago
    > We made approximately the same progress in the past 100 years as the prior 1000 as the prior 30,000 I hear this sort of argument all the time, but what is it even based on? There’s no clear definition of scientific and technological progress, much less something that’s measurable clearly enough to make claims like this. As I understand it, the idea is simply “Ooo, look, it took ten thousand years to go from fire to wheel, but only a couple hundred to go from printing press to airplane!!!”, and I guess that’s true (at least if you have a very juvenile, Sid Meier’s Civilization-like understanding of what history even is) but it’s also nonsense to try and extrapolate actual numbers from it.
    - thegrimmest 1y ago
      Plotting the highest observable assembly index over time will yield an exponential curve starting from the beginning of the universe. This is the closest I’m aware of to a mathematical model quantifying the distinct impression that local complexity has been increasing exponentially.
  - qwertylicious 1y ago
    > Progress has been exponential in the generic. Has it? Really? Consider theoretical physics, which hasn't significantly advancement since the advent of general relativity and quantum theory. Or neurology, where we continue to have only the most basic understanding of how the human mind actually works (let alone the origin of consciousness). Heck, let's look at good ol' Moore's Law, which started off exponential but has slowed down dramatically. It's said that an S curve always starts out looking exponential, and I'd argue in all of those cases we're seeing exactly that. There's no reason to assume technological progress in general, whether via human or artificial intelligence, is necessarily any different.
esafak 1y ago
> If AGI is ever achieved, it would open the door to recursive self improvement ... They are unrelated. All you need is a way for continual improvement without plateauing, and this can start at any level of intelligence. As it did for us; humans were once less intelligent. Using the flagship to bootstrap the next iteration with synthetic data is standard practice now. This was mentioned in the GPT5 presentation. At the rate things are going I think this will get us to ASI, and it's not going to feel epochal for people who have interacted with existing models, but more of the same. After all, the existing models are already smarter than most humans and most people are taking it in their stride. The next revolution is going to be embodiment. I hope we have the commonsense to stop there, before instilling agency.
- andsoitis 1y ago
  > As it did for us; humans were once less intelligent. Do we know what drove the increases in intelligence? Was it some level of intelligence bootstrapping the next level of intelligence? OR was it other biophysical and environmental effects that shaped increasing intelligence?
  - esafak 1y ago
    I've read that social pressures were the primary driver. But robots don't have to take the same path. We're doing the hard work for them... https://www.sciencedirect.com/topics/psychology/social-intelligence-hypothesis https://www.sciencedirect.com/topics/psychology/social-intel...
    - FuriouslyAdrift 1y ago
      Exactly... evolution doesn't select for intelligence. It favors robustness.
  - FabHK 1y ago
    BTW, it appears that the Flynn effect might have reversed recently. US: "A reverse Flynn effect was found for composite ability scores with large US adult sample from 2006 to 2018 and 2011 to 2018. Domain scores of matrix reasoning, letter and number series, verbal reasoning showed evidence of declining scores." https://www.sciencedirect.com/science/article/pii/S0160289623000156 https://www.sciencedirect.com/science/article/pii/S016028962... https://www.forbes.com/sites/michaeltnietzel/2023/03/23/american-iq-test-scores-show-recent-declines-according-to-new-study/ https://www.forbes.com/sites/michaeltnietzel/2023/03/23/amer... Denmark: "The results showed that the estimated mean IQ score increased from a baseline set to 100 (SD: 15) among individuals born in 1940 to 108.9 (SD: 12.2) among individuals born in 1980, since when it has decreased." https://pubmed.ncbi.nlm.nih.gov/34882746/ https://pubmed.ncbi.nlm.nih.gov/34882746/ https://pubmed.ncbi.nlm.nih.gov/34882746/#&gid=article-figures&pid=fig-1-uid-0 https://pubmed.ncbi.nlm.nih.gov/34882746/#&gid=article-figur...
stoneyhrm1 1y ago
> revolutionary breakthroughs in essentially all field This doesn't really make sense outside computers. Since AI would be training itself, it needs to have the right answers, but as of now it doesn't really interact with the physical world. The most it could do is write code, and check things that have no room for interpretation, like speed, latency, percentage of errors, exceptions, etc. But, what other fields would it do this in? How can it makes strives in biology, it can't dissect animals, it can't figure more out about plants that humans feed into the training data. Regarding math, math is human-defined. Humans said "addition does this", "this symbol means that", etc. I just don't understand how AI could ever surpass anything human known before we live by the rules defined by us.
- nickpinkston 1y ago
  [in Morpheus voice] "But when AI got finally access to a bank account and LinkedIn, the machines found the only source of hands it would ever need." That's my bet at least - especially with remote work, etc. is that if the machines were really superhuman, they could convince people to partner with it to do anything else.
  - Earw0rm 1y ago
    You mean like convincing them to invest implausibly huge sums of money in building ever bigger data-centres?
    - nickpinkston 1y ago
      It is interesting that, even before real AGI/ASI gets here, that "the system wants what it wants", like capitalism + computing/internet creates the conditions for an infinite amplification loop. I am amazed, hopeful, and terrified TBH.
      - Earw0rm 1y ago
        Feedback gain loops have a tendency to continue right up to the point they blow a circuit breaker or otherwise drive their operating substrate beyond linear conditions.
solumunus 1y ago
For some reason people pre suppose super intelligence into AGI. What if AGI had diminishing returns around human level intelligence? They still have to deal with all the same knowledge gaps we have.
mycall 1y ago
You are missing the point where synthetic data, deterministic tooling (written by AI) and new discoveries by each model generation feeds into the next model. This iteration is the key to going beyond human intelligence.
andsoitis 1y ago
> So the AI would be improving itself Why would the AI want to improve itself? From whence would that self-motivation stem?
- Earw0rm 1y ago
  At the point where it can even be said to have a self, that battle is mostly won. I am very far from convinced that we are at or near that point.
  - computerthings 1y ago
    [dead]
- keyworkorange 1y ago
  This reminded me of a few subplots in Murderbot- (Do yourself a favor and check it out if you haven't, it's a fun, quick read) But seriously, one would assume there's a reward system of some sort at play, otherwise why do anything?
jltsiren 1y ago
There is no particular reason to assume that recursive self-improvement would be rapid. All the technological revolutions so far have accounted for little more than a 1.5% sustained annual productivity growth. There are always some low-hanging fruit with new technology, but once they have been picked, the effort required for each incremental improvement tends to grow exponentially. That's my default scenario with AGI as well. After AGI arrives, it will leave humans behind very slowly.
- ccozan 1y ago
  Energy. Is the only limiting factor. If a true AGI would emerge, it will immediatelly try to secure energy sources or advances in efficiency. You cannot beat humans with megawatts!
- FabHK 1y ago
  Suppose you don't have a hammer, but just hammer at things with bare hands. Then you find some primitive rock - artificial general hammering! With that you can over time build some primitive hammer - now we're talking superhuman general hammering. With that you can then build a better hammer more quickly, and boom, you have recursive self-improvement, and soon you'll take over the world.
FuriouslyAdrift 1y ago
Math... lots and lots of math solutions. Like if it could figure out the numerical sign problem, it could quite possibly be able to simulate all of physics.
aldousd666 1y ago
Those problems aren't just waiting on smarts/intelligence. Those would require experimentation in the real world. You can't solve chemistry by just thinking about it really hard. You still have to do experiments. A super intelligent machine may be better at coming up with experiments to do than we are, but without the right stuff to do them, it can't 'solve' anything of the like.
p0nce 1y ago
Well it could also self-improve increasingly slowly.
throwawayb2025 1y ago
Recursive improvement without any physical change maybe limited. If any physical change like more gpu or different network configuration is required to experiment and again change to learn from it that might not be easy. Convincing human to do on AGI behalf may not be that simple. There might be multiple path to try and teams may not agree with each other. Specially if the cost of trial is high.
ieee2 1y ago
AI can be trained on some special knowledge of person A and another special knowledge of person B. These two persons may never met before and therefore they can not combine their knowledge to get some new knowledge or insight. AI can do it fine as it knows A and B. And that is knowledge creation.
moron4hire 1y ago
That's only assuming there are no fundamental limits or major barriers to computation. Back a hundred years ago at the dawn of flight, one could have said a very similar thing about aircraft performance. And for a time in the 1950s, it looked like aircraft speed was growing exponentially over time. But there haven't been any new airspeed records (at least, officially recorded) since 1986, because it turns out going Mach 3+ is fairly dangerous and approaching some rather severe materials and propulsion limitations, making it not at all economical. I would also not be surprised if the process of developing something comparable to human intelligence, assuming the extreme computation, energy, and materials issues of packing that much computation and energy into a single system could be overcome, the AI also develops something comparable to human desire and/or mental health issues. There is a not-zero chance we could end up with AI that doesn't want to do what we ask it to do or doesn't work all the time because it wants to do other things. You can't just assume exponential growth is a forgone conclusion.
dabockster 1y ago
> diminishing returns I think this is a hard kick below the belt for anyone trying to develop AGI using current computer science. Current AIs only really generate - no, regenerate text based on their training data. They are only as smart as other data available. Even when an AI "thinks", it's only really still processing existing data rather than making a genuinely new conclusion. It's the best text processor ever created - but it's still just a text processor at its core. And that won't change without more hard computer science being performed by humans. So yeah, I think we're starting to hit the upper limits of what we can do with Transformers technology. I'd be very surprised if someone achieved "AGI" with current tech. And, if it did get achieved, I wouldn't consider it "production ready" until it didn't need a nuclear reactor to power it.
- weebull 1y ago
  Absolutely. All the talk around AGI being some barrier through which unheard of glories can be unlocked sound very much like "perpetual motion machine" talk.
SkyMarshal 1y ago
> But I think we're not even on the path to creating AGI. It seems like the LLM model will be component of an eventual AGI, it's voice per se, but not its mind. The mind still requires another innovation or breakthrough we haven't seen yet.

thinkingtoilet 1y ago

>And, for at least a while, it would also presumably be doing so at an exponentially increasing rate. Why would you presume this? I think part of a lot of people's AI skepticism is talk like this. You have no idea. Full stop. Why wouldn't progress be linear? As new breakthroughs come, newer ones will be harder to come by. Perhaps it's exponential. Perhaps it's linear. No one knows.

robwwilliams 1y ago
Or bottlenecked by data availability just like we humans are. Nothing will be exponential if a loop in the real world of science and engineering is involved.
- jatora 1y ago
  Coding and algorithmic advance does not require real world experimentation.
  - ceh123 1y ago
    Right but for self improving AI, training new models does have a real world bottleneck: energy and hardware. (Even if the data bottleneck is solved too)
- MarcelOlsz 1y ago
  Aren't we bottlenecked by not having any "prior art", as in not having reverse engineered any thinking machine like even a fly's brain? We can't even agree on a definition of consciousness and still don't understand the brain or how it works (to the extent that reverse engineering it can tell us something).
jatora 1y ago
It's a logical presumption. Researchers discover things. AGI is a researcher that can be scaled, research faster, and requires no downtime. Full stop if you dont find that obvious you should probably figure out where your bias is coming from. Coding and algorithmic advance does not require real world experimentation.
- throwaway48476 1y ago
  This is fundamentally flawed. There are upper bounds of efficiency that are laws of nature. To assume AI would be supernatural is magical thinking.
  - solumunus 1y ago
    And yet it seems to be the prevailing opinion even among very smart people. The “singularity” it’s just presumed. I’m highly skeptical to say the least. Look how much energy it’s taking to engineer these models which are still nowhere near AGI. When we get to AGI it won’t be immediately super intelligent and perhaps it never will be. Diminishing returns surely apply to anything that is energy based?
  - me-vs-cat 1y ago
    Natural intelligence appears supernatural from our current understanding, so it's not surprising that AGI also appears so.
    - goatlover 1y ago
      Neither appears supernatural from a scientific understanding.
- thinkingtoilet 1y ago
  Prove it.
- dkersten 1y ago
  If AGI has human capability, why would we think it could research any faster than a human? Sure, you can scale it, but if an LLM takes, say, $1 million a year to run an AGI instance, but it costs only $500k for one human researcher, then it still doesn’t get you anywhere faster than humans do. It might scale up, it might not, we don’t know. We won’t know until we reach it. We also don’t know if it scales linearly. Or if it’s learning capability and capacity will able to support exponential capability increase. Our current LLM’s don’t even have the capability of self improvement or learning even if they were capable: they can accumulate additional knowledge through the context window, but the models are static unless you fine tune or retrain them. What if our current models were ready for AGI but these limitations are stopping it? How would we ever know? Maybe it will be able to self improve but it will I’ll take exponentially larger amounts of training data. Or exponentially larger amounts of energy. Or maybe it can become “smarter” but at the cost of being larger to the point where the laws of physics mean it has to think slower, 2x the thinking but 2x the time, could happen! What if an AGI doesn’t want to improve? Far too many unknowns to say what will happen.
hennell 1y ago
No one knows, but it's a reasonable assumption surely. If you're theorising a AGI, that has recursive self improvement, exponential improvements seem almost unavoidable. AGI improves understanding of electronics, physics etc, that improves the AGI leading to new understandings and so on. Add in that new discoveries in one field might inspire the AGI/humans to find things in others and it seems hard to imagine a situation where theres not a lot of progress everywhere (at least theoretical progress, building new things might be slower / more costly then reasoning they would work.) Where I'm skeptical of AI would be in the idea an LLM can ever get to AGI level, if AGI is even really possible, and if the whole thing is actually viable. I'm also very skeptical that the discoveries of any AGI would be shared in ways that would allow exponential growth; licenses stopping using their AGI to make your own, copyright on the new laws of physics and royalties on any discovery you make from using those new laws etc.
- thinkingtoilet 1y ago
  >If you're theorising a AGI, that has recursive self improvement, exponential improvements seem almost unavoidable. Prove it. Also, AI will need resources. Hardware. Water. Electricity. Can those resources be supplied at an exponential rate? People need to calm down and stop stating things as truth when they literally have no idea.
  - johnisgood 1y ago
    Which is why I am an agnostic. :)
  - mangodrunk 1y ago
    Well said. It does seem that many who speculate on this are not taking into account limits that more/faster processing won’t actually help much. Say an algorithm is proven to be O(n!) for all cases, at a certain size of n, there’s not much that can be done if the algorithm is needed as is.
- deleted 1y ago
  [deleted]
frognumber 1y ago
I always consider different options when planning for the future, but I'll give the argument for exponential: Progress has been exponential in the generic. We made approximately the same progress in the past 100 years as the prior 1000 as the prior 30,000, as the prior million, and so on, all the way back to multicellular life evolving over 2 billion years or so. There's a question of the exponent, though. Living through that exponential growth circa 50AD felt at best linear, if not flat.
- FabHK 1y ago
  So you concede that there's nothing special about AI versus earlier innovations?
- ileonichwiesz 1y ago
  > We made approximately the same progress in the past 100 years as the prior 1000 as the prior 30,000 I hear this sort of argument all the time, but what is it even based on? There’s no clear definition of scientific and technological progress, much less something that’s measurable clearly enough to make claims like this. As I understand it, the idea is simply “Ooo, look, it took ten thousand years to go from fire to wheel, but only a couple hundred to go from printing press to airplane!!!”, and I guess that’s true (at least if you have a very juvenile, Sid Meier’s Civilization-like understanding of what history even is) but it’s also nonsense to try and extrapolate actual numbers from it.
  - thegrimmest 1y ago
    Plotting the highest observable assembly index over time will yield an exponential curve starting from the beginning of the universe. This is the closest I’m aware of to a mathematical model quantifying the distinct impression that local complexity has been increasing exponentially.
- qwertylicious 1y ago
  > Progress has been exponential in the generic. Has it? Really? Consider theoretical physics, which hasn't significantly advancement since the advent of general relativity and quantum theory. Or neurology, where we continue to have only the most basic understanding of how the human mind actually works (let alone the origin of consciousness). Heck, let's look at good ol' Moore's Law, which started off exponential but has slowed down dramatically. It's said that an S curve always starts out looking exponential, and I'd argue in all of those cases we're seeing exactly that. There's no reason to assume technological progress in general, whether via human or artificial intelligence, is necessarily any different.

esafak 1y ago

> If AGI is ever achieved, it would open the door to recursive self improvement ... They are unrelated. All you need is a way for continual improvement without plateauing, and this can start at any level of intelligence. As it did for us; humans were once less intelligent. Using the flagship to bootstrap the next iteration with synthetic data is standard practice now. This was mentioned in the GPT5 presentation. At the rate things are going I think this will get us to ASI, and it's not going to feel epochal for people who have interacted with existing models, but more of the same. After all, the existing models are already smarter than most humans and most people are taking it in their stride. The next revolution is going to be embodiment. I hope we have the commonsense to stop there, before instilling agency.

andsoitis 1y ago
> As it did for us; humans were once less intelligent. Do we know what drove the increases in intelligence? Was it some level of intelligence bootstrapping the next level of intelligence? OR was it other biophysical and environmental effects that shaped increasing intelligence?
- esafak 1y ago
  I've read that social pressures were the primary driver. But robots don't have to take the same path. We're doing the hard work for them... https://www.sciencedirect.com/topics/psychology/social-intelligence-hypothesis https://www.sciencedirect.com/topics/psychology/social-intel...
  - FuriouslyAdrift 1y ago
    Exactly... evolution doesn't select for intelligence. It favors robustness.
- FabHK 1y ago
  BTW, it appears that the Flynn effect might have reversed recently. US: "A reverse Flynn effect was found for composite ability scores with large US adult sample from 2006 to 2018 and 2011 to 2018. Domain scores of matrix reasoning, letter and number series, verbal reasoning showed evidence of declining scores." https://www.sciencedirect.com/science/article/pii/S0160289623000156 https://www.sciencedirect.com/science/article/pii/S016028962... https://www.forbes.com/sites/michaeltnietzel/2023/03/23/american-iq-test-scores-show-recent-declines-according-to-new-study/ https://www.forbes.com/sites/michaeltnietzel/2023/03/23/amer... Denmark: "The results showed that the estimated mean IQ score increased from a baseline set to 100 (SD: 15) among individuals born in 1940 to 108.9 (SD: 12.2) among individuals born in 1980, since when it has decreased." https://pubmed.ncbi.nlm.nih.gov/34882746/ https://pubmed.ncbi.nlm.nih.gov/34882746/ https://pubmed.ncbi.nlm.nih.gov/34882746/#&gid=article-figures&pid=fig-1-uid-0 https://pubmed.ncbi.nlm.nih.gov/34882746/#&gid=article-figur...

stoneyhrm1 1y ago

> revolutionary breakthroughs in essentially all field This doesn't really make sense outside computers. Since AI would be training itself, it needs to have the right answers, but as of now it doesn't really interact with the physical world. The most it could do is write code, and check things that have no room for interpretation, like speed, latency, percentage of errors, exceptions, etc. But, what other fields would it do this in? How can it makes strives in biology, it can't dissect animals, it can't figure more out about plants that humans feed into the training data. Regarding math, math is human-defined. Humans said "addition does this", "this symbol means that", etc. I just don't understand how AI could ever surpass anything human known before we live by the rules defined by us.

nickpinkston 1y ago
[in Morpheus voice] "But when AI got finally access to a bank account and LinkedIn, the machines found the only source of hands it would ever need." That's my bet at least - especially with remote work, etc. is that if the machines were really superhuman, they could convince people to partner with it to do anything else.
- Earw0rm 1y ago
  You mean like convincing them to invest implausibly huge sums of money in building ever bigger data-centres?
  - nickpinkston 1y ago
    It is interesting that, even before real AGI/ASI gets here, that "the system wants what it wants", like capitalism + computing/internet creates the conditions for an infinite amplification loop. I am amazed, hopeful, and terrified TBH.
    - Earw0rm 1y ago
      Feedback gain loops have a tendency to continue right up to the point they blow a circuit breaker or otherwise drive their operating substrate beyond linear conditions.

solumunus 1y ago

For some reason people pre suppose super intelligence into AGI. What if AGI had diminishing returns around human level intelligence? They still have to deal with all the same knowledge gaps we have.

mycall 1y ago

You are missing the point where synthetic data, deterministic tooling (written by AI) and new discoveries by each model generation feeds into the next model. This iteration is the key to going beyond human intelligence.

andsoitis 1y ago

> So the AI would be improving itself Why would the AI want to improve itself? From whence would that self-motivation stem?

Earw0rm 1y ago
At the point where it can even be said to have a self, that battle is mostly won. I am very far from convinced that we are at or near that point.
- computerthings 1y ago
  [dead]
keyworkorange 1y ago
This reminded me of a few subplots in Murderbot- (Do yourself a favor and check it out if you haven't, it's a fun, quick read) But seriously, one would assume there's a reward system of some sort at play, otherwise why do anything?

jltsiren 1y ago

There is no particular reason to assume that recursive self-improvement would be rapid. All the technological revolutions so far have accounted for little more than a 1.5% sustained annual productivity growth. There are always some low-hanging fruit with new technology, but once they have been picked, the effort required for each incremental improvement tends to grow exponentially. That's my default scenario with AGI as well. After AGI arrives, it will leave humans behind very slowly.

ccozan 1y ago
Energy. Is the only limiting factor. If a true AGI would emerge, it will immediatelly try to secure energy sources or advances in efficiency. You cannot beat humans with megawatts!
FabHK 1y ago
Suppose you don't have a hammer, but just hammer at things with bare hands. Then you find some primitive rock - artificial general hammering! With that you can over time build some primitive hammer - now we're talking superhuman general hammering. With that you can then build a better hammer more quickly, and boom, you have recursive self-improvement, and soon you'll take over the world.

FuriouslyAdrift 1y ago

Math... lots and lots of math solutions. Like if it could figure out the numerical sign problem, it could quite possibly be able to simulate all of physics.

aldousd666 1y ago

Those problems aren't just waiting on smarts/intelligence. Those would require experimentation in the real world. You can't solve chemistry by just thinking about it really hard. You still have to do experiments. A super intelligent machine may be better at coming up with experiments to do than we are, but without the right stuff to do them, it can't 'solve' anything of the like.

p0nce 1y ago

Well it could also self-improve increasingly slowly.

throwawayb2025 1y ago

Recursive improvement without any physical change maybe limited. If any physical change like more gpu or different network configuration is required to experiment and again change to learn from it that might not be easy. Convincing human to do on AGI behalf may not be that simple. There might be multiple path to try and teams may not agree with each other. Specially if the cost of trial is high.

ieee2 1y ago

AI can be trained on some special knowledge of person A and another special knowledge of person B. These two persons may never met before and therefore they can not combine their knowledge to get some new knowledge or insight. AI can do it fine as it knows A and B. And that is knowledge creation.

moron4hire 1y ago

That's only assuming there are no fundamental limits or major barriers to computation. Back a hundred years ago at the dawn of flight, one could have said a very similar thing about aircraft performance. And for a time in the 1950s, it looked like aircraft speed was growing exponentially over time. But there haven't been any new airspeed records (at least, officially recorded) since 1986, because it turns out going Mach 3+ is fairly dangerous and approaching some rather severe materials and propulsion limitations, making it not at all economical. I would also not be surprised if the process of developing something comparable to human intelligence, assuming the extreme computation, energy, and materials issues of packing that much computation and energy into a single system could be overcome, the AI also develops something comparable to human desire and/or mental health issues. There is a not-zero chance we could end up with AI that doesn't want to do what we ask it to do or doesn't work all the time because it wants to do other things. You can't just assume exponential growth is a forgone conclusion.

dabockster 1y ago

> diminishing returns I think this is a hard kick below the belt for anyone trying to develop AGI using current computer science. Current AIs only really generate - no, regenerate text based on their training data. They are only as smart as other data available. Even when an AI "thinks", it's only really still processing existing data rather than making a genuinely new conclusion. It's the best text processor ever created - but it's still just a text processor at its core. And that won't change without more hard computer science being performed by humans. So yeah, I think we're starting to hit the upper limits of what we can do with Transformers technology. I'd be very surprised if someone achieved "AGI" with current tech. And, if it did get achieved, I wouldn't consider it "production ready" until it didn't need a nuclear reactor to power it.

weebull 1y ago
Absolutely. All the talk around AGI being some barrier through which unheard of glories can be unlocked sound very much like "perpetual motion machine" talk.

SkyMarshal 1y ago

> But I think we're not even on the path to creating AGI. It seems like the LLM model will be component of an eventual AGI, it's voice per se, but not its mind. The mind still requires another innovation or breakthrough we haven't seen yet.