6 ms·

> But our models make it clear that such an [intelligence] explosion may not follow if there are diminishing returns (“ideas become harder to find”) or if feedb

> But our models make it clear that such an [intelligence] explosion may not follow if there are diminishing returns (“ideas become harder to find”) or if feedback loops become bottlenecked. How is this not obvious to everyone? As we advance it becomes more difficult to advance. You obviously make most advancements around the things that are easiest to improve. Then all the easy things are done. So you go onto the next easiest things. They're "the easy things" from that standpoint but that doesn't mean they aren't harder than "the easy things" when you started. Complexity increases as precision increases.

EarlKing 3mo ago
> How is this not obvious to everyone? Because everyone's thinking around intelligence is incredibly muddled by a variety of factors, and no one is particularly motivated to actually correct anyone's mistaken notions on the matter.
- godelski 3mo ago
  I'm conflicted. On one hand I think we should more openly call people idiots and push back. On the other hand there's Descartes argument for idiots in good company. I just wish all the people that claimed to care only about truth would actually care about truth. Feels like society is more that trope where someone says a joke to a crowd and no one hears it except one charismatic person who repeats it and gets all the laughs. In reality it feels like the repeated version of the joke doesn't even make sense, it is just vibes.
red75prime 3mo ago
> How is this not obvious to everyone? That RSI can be bottlenecked? I guess this is obvious to many people. Whether RSI will be bottlenecked (at some not very interesting stage) is another question.
- RandomLensman 3mo ago
  Why would it not be bottlenecked? Is intelligence so valuable as such?
  - ben_w 3mo ago
    Our intelligence was enough to turn some balding apes into atom-bomb wielders and astronauts over a few thousand generations. Consider a hypothetical: it is possible to make a semiconductor clone of a human brain, that runs at semiconductor speeds, has semiconductor size-scale, and has a power requirement of the Landauer limit. This copy would be smaller than a pea, and you'd get to pick on a sliding scale between "same power draw as human but thinks faster than us to the same multiplier that we jog faster than continental drift" or "thinks as fast as we do while using around a few µW of power". We do not know how to do this. We don't know how far we are from figuring out how to do this. We do know* evolution made our brains, and we do use simulated evolution as a standard technique in machine learning. But it may well be that just as no human knows how to design a human mind, we find the best AI we can make don't know how to make a better AI, at which point they're stumbling blind in the dark: while evolution is a neat method, it is limited, blind to what the best next step is at any moment. * well, those of us who are not creationists, at least.
    - RandomLensman 3mo ago
      Is it possible to clone without destroying the original brain (and is a semiconductor the right substrate for it)? Thinking of the no cloning theorem here...
      - ben_w 3mo ago
        The no cloning theorem applies to quantum states; when it comes to information processed by it, it's unlikely that a big wet messy hot thing like a cell is usefully treated as "a quantum state", let alone an entire brain. However, this is irrelevant to the thought experiment, for which it is sufficient to merely be as competent as a human, rather than a completely quantum-perfect replica. Clone in the sense you would use the photoshop brush, or perhaps in the sense you might be sued for trademark infringement.
vidarh 3mo ago
Because they depend on whether the rate of improvement of self-improvement outpaces the rate of increase in difficulty or not, and at some points they clearly do - e.g. a lot of skills makes the relative rate of subsequent improvement easier for a while. It may seem obvious that it can't last, but showing the conditions where it can't still matters.
setgree 3mo ago
The goal of a modeling exercise like this, which you don’t have to buy, is to generate a simple set of initial conditions that can explain things we already know. Then, we can manipulate some initial parameter value to make predictions about things we don’t see, but might. Likewise, it is obvious that gravity exists, but a simple model that explains where it comes from (in quantum terms) would be a big breakthrough iff it came with plausibly testable implications that could be tested via experiment.
soulofmischief 3mo ago
Progress is often lumpy, modulated by new discoveries and their applications.
tossandthrow 3mo ago
> As we advance it becomes more difficult to advance I don't think this follows. We have advanced tremendously over the past 200 years, and we are likely going into a time with rapid advancement again. With advancement, we also develop tools (eg. Llms) that assist advancing.
- Windchaser 3mo ago
  To put it another way, the cost per advancement increases. That’s “as we advance it becomes more difficult to advance”. However, because of the prior advances, you also have more resources to throw at future advancements. So, then, the question is whether the “profits” on the last advance are enough to pay for the next one. We can define a new term, “affordability”, as what % “profit” you can expect from each advance relative to its cost, telling us whether it becomes relatively easier or harder to continue to advance.
  - tossandthrow 3mo ago
    You could argue that there is a capital built up - but even that is difficult as a lot of the knowledge for, eg., building LLMs can not demand rent. Everybody can build their own transformer networks. > To put it another way, the cost per advancement increases. This is not true when you don't use capital that demands rent. On the contrary, the cost per advancement is actually decreasing as we develop more knowledge. LLMs are a cognitive technology (as contrasted to a communication technology). And it will help us tremendously manage knowledge such that you can utilize it better decreasing the cost of advancement.
    - RandomLensman 3mo ago
      Whether or not it decreases the costs of advancedment would depend on what is causing advancement to have costs. Some things might be addressable by cognitive technologies, others might not. If capital doesn't create a return evenutally, why would it invest? (Or did you mean "rent" in the narrow economics definition? But then not sure how that applies here).
- 3mo ago
exe34 3mo ago
> How is this not obvious to everyone? Probably because it's not true. We had shitty neural networks for decades before the recent explosion. That particular branch may be a dead end, but there could be others lurking and waiting for their time.
deleted 3mo ago
[deleted]
bryanrasmussen 3mo ago
>How is this not obvious to everyone? because people have other ideas https://en.wikipedia.org/wiki/Technological_singularity https://en.wikipedia.org/wiki/Technological_singularity
- WJW 3mo ago
  People have other dreams, and have written about them in science fiction. But science fiction can just handwave away resource problems that exist in real life so that the plot can happen. Just as the popular "grey goo" nanomachine disaster can easily be shown to be impossible due to resource imbalances and energy shortages, AI recursive self improvements rapidly slows down due to problems with complexity, training data and compute availability (whether due to actual processors or just due to energy demands).
  - bryanrasmussen 3mo ago
    whether or not the singularity would be best thought of as science fiction, it was not presented as such, and has not been taken as such by a great number of people, and has formed the basis of a lot of opinions as to how things will pan out. It is these opinions that make the conclusion the parent poster supposed should be obvious to everyone not obvious to so many.
  - ben_w 3mo ago
    > Just as the popular "grey goo" nanomachine disaster can easily be shown to be impossible due to resource imbalances and energy shortages, Depends on specifics and how dramatic they're being, given that "mold infestation" can come in grey.
  - godelski 3mo ago
    The problem is in cases of the singularity it's being treated like reality rather than fiction
- Majromax 3mo ago
  > because people have other ideas https://en.wikipedia.org/wiki/Technological_singularity https://en.wikipedia.org/wiki/Technological_singularity In a weak sense, singularities are common and should be expected every so often. In a mathematical sense, a singularity is where a model 'blows up' and neglected terms become part of the dominant balance. Industrialization hit a point of diminishing returns, but the industrial revolution was nonetheless a 'singularity,' where life afterwards was qualitatively unpredictable to people who lived before. Likewise, agriculture was such a technological singularity to hunter-gatherer ancestors. I could even make a decent argument that writing and literacy were such a singularity, making inconceivable social organizations routine. In that weak sense I expect AI to be a singularity, recursive self improvement or no. Life in 2050 may be completely unpredictable to someone who was taken out of time in the year 2000. The remaining questions are speed and intensity, and both of these questions are related to RSI. If RSI works, then the 'fast takeoff' visions become more plausible where society transforms over months to a few years – at least locally where the enabling technologies have diffused. If not, it might take a couple of decades.
FrustratedMonky 3mo ago
"How is this not obvious to everyone? As we advance it becomes more difficult to advance. " It's a type of recent change blindness. Because for the last few years there are seemingly impossible breakthroughs every few months. Literally, things everyone said would take 30 or 100 years, happen within months. It is really easy to think this will continue. The new normal. Really, we haven't seen it slow down at all, so why would we expect the miracles to stop all of a sudden? It seems like accelerating, and decelerating, are both fair game as future directions at this point. So it is nice that the paper put some weight behind the argument that this could all grind to a halt for a lot of different reasons beyond the hype.
- breuleux 3mo ago
  > Literally, things everyone said would take 30 or 100 years, happen within months. "Everyone"? Plenty of people in the 50s said we'd have sentient humanoid robots before the year 2000. And flying cars. Sure, many people underestimate the rate of change. But techno-optimists have consistently overestimated it, and they still do.
  - FrustratedMonky 3mo ago
    I was sloppy. I didn't mean to include science fiction authors, or extreme futurists. "everyone", being, a typical average of realistic predictions. In the 90's. I remember people talking about voice recognition will take 50 years. Let alone understanding. Now this is an easy base line feature. Let alone protein folding.
    - breuleux 3mo ago
      > I didn't mean to include science fiction authors, or extreme futurists. Science fiction authors are quite influential, and I don't think they were the only ones either. I expect that plenty of actual scientists had similar views. It wasn't extreme. > In the 90's. I remember people talking about voice recognition will take 50 years. Sure. Who? How many people said that? How many people said the opposite? If we're doing anecdotes, I remember that circa 2010 it was a fairly normal belief that fully self-driving cars would be widespread by 2020. I mean, I thought they would. Point is, many people say many different things, there's never been any widespread agreement about what the future holds. They underestimate, they overestimate, but mostly reality moves sideways in directions nobody thought of and which are somehow simultaneously way more impressive and way less impressive than our expectations.
jamesrcole 3mo ago
> As we advance it becomes more difficult to advance. You obviously make most advancements around the things that are easiest to improve. Then all the easy things are done. This isn't some foregone conclusion. It completely depends on the rate at which the intelligence and abilities of the AI increases. If that rate was high enough, then the harder and harder problems would become easier and easier for it.
- mpalmer 3mo ago
  Just like fully extending a ladder brings the moon closer.
- dinfinity 3mo ago
  Actually, evolution seems to show the opposite: The rate of advancement has only sped up, with billions of years between significant changes going to millions, to thousands, to tens and arguably to mere years now. Having said that, we're probably looking at an S-curve with the physical limits of reality getting in the way in the end.
  - godelski 3mo ago
    You're ignoring a lot of important factors. I'll give you most obvious ones. We often do genetics testing on fruit flies because they reproduce quickly and the initial population is large. Your claim ignores this fact. The base population and number of interactions has increased.
    - dinfinity 3mo ago
      I am not ignoring anything, I'm looking at the broader picture, which includes non-biological evolution. Simple rebuttal to your specific point: The population of self-improving AIs will also go from 0 to many more. In a broader sense evolution moved from very static simple domains to dynamic malleable complex domains. Biological evolution speed is glacial compared to cultural evolution speed. Even then, cultural evolution is fairly slow compared to technological evolution.
  - 3mo ago

EarlKing 3mo ago

> How is this not obvious to everyone? Because everyone's thinking around intelligence is incredibly muddled by a variety of factors, and no one is particularly motivated to actually correct anyone's mistaken notions on the matter.

godelski 3mo ago
I'm conflicted. On one hand I think we should more openly call people idiots and push back. On the other hand there's Descartes argument for idiots in good company. I just wish all the people that claimed to care only about truth would actually care about truth. Feels like society is more that trope where someone says a joke to a crowd and no one hears it except one charismatic person who repeats it and gets all the laughs. In reality it feels like the repeated version of the joke doesn't even make sense, it is just vibes.

red75prime 3mo ago

> How is this not obvious to everyone? That RSI can be bottlenecked? I guess this is obvious to many people. Whether RSI will be bottlenecked (at some not very interesting stage) is another question.

RandomLensman 3mo ago
Why would it not be bottlenecked? Is intelligence so valuable as such?
- ben_w 3mo ago
  Our intelligence was enough to turn some balding apes into atom-bomb wielders and astronauts over a few thousand generations. Consider a hypothetical: it is possible to make a semiconductor clone of a human brain, that runs at semiconductor speeds, has semiconductor size-scale, and has a power requirement of the Landauer limit. This copy would be smaller than a pea, and you'd get to pick on a sliding scale between "same power draw as human but thinks faster than us to the same multiplier that we jog faster than continental drift" or "thinks as fast as we do while using around a few µW of power". We do not know how to do this. We don't know how far we are from figuring out how to do this. We do know* evolution made our brains, and we do use simulated evolution as a standard technique in machine learning. But it may well be that just as no human knows how to design a human mind, we find the best AI we can make don't know how to make a better AI, at which point they're stumbling blind in the dark: while evolution is a neat method, it is limited, blind to what the best next step is at any moment. * well, those of us who are not creationists, at least.
  - RandomLensman 3mo ago
    Is it possible to clone without destroying the original brain (and is a semiconductor the right substrate for it)? Thinking of the no cloning theorem here...
    - ben_w 3mo ago
      The no cloning theorem applies to quantum states; when it comes to information processed by it, it's unlikely that a big wet messy hot thing like a cell is usefully treated as "a quantum state", let alone an entire brain. However, this is irrelevant to the thought experiment, for which it is sufficient to merely be as competent as a human, rather than a completely quantum-perfect replica. Clone in the sense you would use the photoshop brush, or perhaps in the sense you might be sued for trademark infringement.

vidarh 3mo ago

Because they depend on whether the rate of improvement of self-improvement outpaces the rate of increase in difficulty or not, and at some points they clearly do - e.g. a lot of skills makes the relative rate of subsequent improvement easier for a while. It may seem obvious that it can't last, but showing the conditions where it can't still matters.

setgree 3mo ago

The goal of a modeling exercise like this, which you don’t have to buy, is to generate a simple set of initial conditions that can explain things we already know. Then, we can manipulate some initial parameter value to make predictions about things we don’t see, but might. Likewise, it is obvious that gravity exists, but a simple model that explains where it comes from (in quantum terms) would be a big breakthrough iff it came with plausibly testable implications that could be tested via experiment.

soulofmischief 3mo ago

Progress is often lumpy, modulated by new discoveries and their applications.

tossandthrow 3mo ago

> As we advance it becomes more difficult to advance I don't think this follows. We have advanced tremendously over the past 200 years, and we are likely going into a time with rapid advancement again. With advancement, we also develop tools (eg. Llms) that assist advancing.

Windchaser 3mo ago
To put it another way, the cost per advancement increases. That’s “as we advance it becomes more difficult to advance”. However, because of the prior advances, you also have more resources to throw at future advancements. So, then, the question is whether the “profits” on the last advance are enough to pay for the next one. We can define a new term, “affordability”, as what % “profit” you can expect from each advance relative to its cost, telling us whether it becomes relatively easier or harder to continue to advance.
- tossandthrow 3mo ago
  You could argue that there is a capital built up - but even that is difficult as a lot of the knowledge for, eg., building LLMs can not demand rent. Everybody can build their own transformer networks. > To put it another way, the cost per advancement increases. This is not true when you don't use capital that demands rent. On the contrary, the cost per advancement is actually decreasing as we develop more knowledge. LLMs are a cognitive technology (as contrasted to a communication technology). And it will help us tremendously manage knowledge such that you can utilize it better decreasing the cost of advancement.
  - RandomLensman 3mo ago
    Whether or not it decreases the costs of advancedment would depend on what is causing advancement to have costs. Some things might be addressable by cognitive technologies, others might not. If capital doesn't create a return evenutally, why would it invest? (Or did you mean "rent" in the narrow economics definition? But then not sure how that applies here).
3mo ago

exe34 3mo ago

> How is this not obvious to everyone? Probably because it's not true. We had shitty neural networks for decades before the recent explosion. That particular branch may be a dead end, but there could be others lurking and waiting for their time.

deleted 3mo ago

[deleted]

bryanrasmussen 3mo ago

>How is this not obvious to everyone? because people have other ideas https://en.wikipedia.org/wiki/Technological_singularity https://en.wikipedia.org/wiki/Technological_singularity

WJW 3mo ago
People have other dreams, and have written about them in science fiction. But science fiction can just handwave away resource problems that exist in real life so that the plot can happen. Just as the popular "grey goo" nanomachine disaster can easily be shown to be impossible due to resource imbalances and energy shortages, AI recursive self improvements rapidly slows down due to problems with complexity, training data and compute availability (whether due to actual processors or just due to energy demands).
- bryanrasmussen 3mo ago
  whether or not the singularity would be best thought of as science fiction, it was not presented as such, and has not been taken as such by a great number of people, and has formed the basis of a lot of opinions as to how things will pan out. It is these opinions that make the conclusion the parent poster supposed should be obvious to everyone not obvious to so many.
- ben_w 3mo ago
  > Just as the popular "grey goo" nanomachine disaster can easily be shown to be impossible due to resource imbalances and energy shortages, Depends on specifics and how dramatic they're being, given that "mold infestation" can come in grey.
- godelski 3mo ago
  The problem is in cases of the singularity it's being treated like reality rather than fiction
Majromax 3mo ago
> because people have other ideas https://en.wikipedia.org/wiki/Technological_singularity https://en.wikipedia.org/wiki/Technological_singularity In a weak sense, singularities are common and should be expected every so often. In a mathematical sense, a singularity is where a model 'blows up' and neglected terms become part of the dominant balance. Industrialization hit a point of diminishing returns, but the industrial revolution was nonetheless a 'singularity,' where life afterwards was qualitatively unpredictable to people who lived before. Likewise, agriculture was such a technological singularity to hunter-gatherer ancestors. I could even make a decent argument that writing and literacy were such a singularity, making inconceivable social organizations routine. In that weak sense I expect AI to be a singularity, recursive self improvement or no. Life in 2050 may be completely unpredictable to someone who was taken out of time in the year 2000. The remaining questions are speed and intensity, and both of these questions are related to RSI. If RSI works, then the 'fast takeoff' visions become more plausible where society transforms over months to a few years – at least locally where the enabling technologies have diffused. If not, it might take a couple of decades.

FrustratedMonky 3mo ago

"How is this not obvious to everyone? As we advance it becomes more difficult to advance. " It's a type of recent change blindness. Because for the last few years there are seemingly impossible breakthroughs every few months. Literally, things everyone said would take 30 or 100 years, happen within months. It is really easy to think this will continue. The new normal. Really, we haven't seen it slow down at all, so why would we expect the miracles to stop all of a sudden? It seems like accelerating, and decelerating, are both fair game as future directions at this point. So it is nice that the paper put some weight behind the argument that this could all grind to a halt for a lot of different reasons beyond the hype.

breuleux 3mo ago
> Literally, things everyone said would take 30 or 100 years, happen within months. "Everyone"? Plenty of people in the 50s said we'd have sentient humanoid robots before the year 2000. And flying cars. Sure, many people underestimate the rate of change. But techno-optimists have consistently overestimated it, and they still do.
- FrustratedMonky 3mo ago
  I was sloppy. I didn't mean to include science fiction authors, or extreme futurists. "everyone", being, a typical average of realistic predictions. In the 90's. I remember people talking about voice recognition will take 50 years. Let alone understanding. Now this is an easy base line feature. Let alone protein folding.
  - breuleux 3mo ago
    > I didn't mean to include science fiction authors, or extreme futurists. Science fiction authors are quite influential, and I don't think they were the only ones either. I expect that plenty of actual scientists had similar views. It wasn't extreme. > In the 90's. I remember people talking about voice recognition will take 50 years. Sure. Who? How many people said that? How many people said the opposite? If we're doing anecdotes, I remember that circa 2010 it was a fairly normal belief that fully self-driving cars would be widespread by 2020. I mean, I thought they would. Point is, many people say many different things, there's never been any widespread agreement about what the future holds. They underestimate, they overestimate, but mostly reality moves sideways in directions nobody thought of and which are somehow simultaneously way more impressive and way less impressive than our expectations.

jamesrcole 3mo ago

> As we advance it becomes more difficult to advance. You obviously make most advancements around the things that are easiest to improve. Then all the easy things are done. This isn't some foregone conclusion. It completely depends on the rate at which the intelligence and abilities of the AI increases. If that rate was high enough, then the harder and harder problems would become easier and easier for it.

mpalmer 3mo ago
Just like fully extending a ladder brings the moon closer.
dinfinity 3mo ago
Actually, evolution seems to show the opposite: The rate of advancement has only sped up, with billions of years between significant changes going to millions, to thousands, to tens and arguably to mere years now. Having said that, we're probably looking at an S-curve with the physical limits of reality getting in the way in the end.
- godelski 3mo ago
  You're ignoring a lot of important factors. I'll give you most obvious ones. We often do genetics testing on fruit flies because they reproduce quickly and the initial population is large. Your claim ignores this fact. The base population and number of interactions has increased.
  - dinfinity 3mo ago
    I am not ignoring anything, I'm looking at the broader picture, which includes non-biological evolution. Simple rebuttal to your specific point: The population of self-improving AIs will also go from 0 to many more. In a broader sense evolution moved from very static simple domains to dynamic malleable complex domains. Biological evolution speed is glacial compared to cultural evolution speed. Even then, cultural evolution is fairly slow compared to technological evolution.
- 3mo ago