4 ms·

RSI is a fetishistic term among the singularity crowd, who imagine AI "recursively" improving itself in some exponential fashion until there is a bright flash o

RSI is a fetishistic term among the singularity crowd, who imagine AI "recursively" improving itself in some exponential fashion until there is a bright flash of white light and it reveals itself in the form of god. Or something like that. I don't know why whoever coined the term chose "recursive" rather than "iterative" - just sounds more likely to lead to infinite regress I suppose. This notion of recursive/iterative self-improvement, whereby generation #1 AI improves itself to create generation #2, then generation #2 further improves itself to create generation #3, etc, seems to conflict with the reality that what we have with LLMs is models whose performance/capability is defined by data, not code, so the most you can do is have your LLM design synthetic data, or just do Karpathy-style "auto research" where all you are doing is using the LLM to automate your experiments. At the end of the day, each experiment, designed by a person and/or LLM, then needs to compete with all your other ideas for compute to be tested at scale, and no amount of recursion or self-improvement will materialize an infinite amount of compute out of thin air, so your recursively synthetic-data gobbling LLM will continue to improve at the same pace it ever did.

jazzyjackson 1mo ago
Yes the exponential self improvement folks have never heard of an eigenvalue I guess. You can loop forever using output as input but at some point the result will stop changing (depending on the function)
- fuzzfactor 1mo ago
  >AI "recursively" improving itself in some exponential fashion until there is a bright flash of white light Sounds like repetitive stress to me. >loop forever using output as input but at some point the result will stop changing Running in place will eventually wear you out too. Plus with some things it can be difficult to know for sure if that's where you are at the time. Even worse may be if you were almost running in place, it could be orders of magnitude more difficult to discern, especially if the scale was massive to an unprecedented degree.
- ajkjk 1mo ago
  that's not really how eigenvalues work... they specifically also model the case where the result keeps changing exponentially.
  - mjburgess 1mo ago
    The claim is that the RSI operation is just finding a fixed point of improvement, RSI(LLM) = RSI(LLM) -- for an optimal LLM* which is a fixed point of RSI As for eigenvalues/vectors, they're fixed points of (1/val)A or A*val
    - ekidd 1mo ago
      Eigenvectors represent fixed directions, not fixed magnitudes. From Wikipedia: > More precisely, an eigenvector v of a linear transformation T is scaled by a constant factor lambda when the linear transformation is applied to it: Tv = lambda v . In other words, repeated multiplication of an eigenvector by a matrix can still create exponential growth.
      - mjburgess 1mo ago
        > of (1/val)A
- marcosdumay 1mo ago
  The name you are looking for is "fixed points", not "eingevalues".
  - HarHarVeryFunny 1mo ago
    The "You can loop forever using output as input but at some point the result will stop changing" idea seems more related to dynamics, feedback and attractor states. If you have a feedback loop of "recursively improving" by generating synthetic data, to train on, to generate more synthetic data, etc, then this is indeed a matter of looping "using output as input", and without any other system inputs it would indeed "converge" to some attractor states.
- HarHarVeryFunny 1mo ago
  I think the limit of what can be achieved with RL and synthetic data generation is better simply described as a leveling off of gains as you extract all the intelligence and knowledge from the original human training data. Of course things will change at some point in the future as we go beyond LLMs, to build creative intelligence not just imitative/predictive intelligence, but right now these companies are stuck in this loop of building synthetic data and RLVR training from that, which means they are essentially building the "generative closure" of the original human training data - trying to squeeze all the juice out of it. To go beyond this they need to add creativity of some sort to generate data that is not ultimately based on the original human training data. They could try something like brute force search (cf agent swarms/graphs), but this is just a more thorough way of exploring the search space defined by the training data - it may find you the "move 37" or low-hanging mathematical proof, but as Demis Hassabis has said, the goal of AGI is not to find move 37 but rather to create something capable of inventing as compelling a game as Go in the first place.
red75prime 1mo ago
What will prevent LLMs from designing robot control circuitry and participating in increase of chip production/design and physical experimentation? How do you think why there's this fad of producing general purpose humanoid robots?
- HarHarVeryFunny 1mo ago
  > What will prevent LLMs from designing robot control circuitry and participating in increase of chip production/design and physical experimentation? Money, regulations, EUV machine lead-times, global helium supply, reality ... It's funny that we've got the Dwarkesh contingent saying that GPUs will become infinitely expensive, and now another contingent saying that they will become infinitely abundant. Even if compute were free, and/or the AI was so smart that it picked the right experiments to run every time ("make no mistakes"), you still have to actually train the model, which takes months, and if model Ver. N+1 depends on model Ver. N, then it's iterative regardless of how much compute you have.
  - red75prime 1mo ago
    Who's saying that compute will become infinitely abundant? "Singularity" is just a way of saying that known models begin to give absurd predictions. Anyway, intelligence is a way of overcoming obstacles. 10 million tonnes of helium is a nice head start and retraining models from scratch is not guaranteed to last forever.
    - HarHarVeryFunny 1mo ago
      > 10 million tonnes of helium is a nice head start Yeah, but then you need to refine it to 99.9999% purity, to be able to use it.
    - HarHarVeryFunny 1mo ago
      AFAIK the notion of a/the technological "singularity" is a point in time where technology is building upon itself (RSI!) so fast, at an ever increasing pace, that the speed of change effectively becomes infinite and incomprehensible to humans. The word "singularity" is presumably coming from math or space, like a black hole singularity where matter becomes infinitely dense and the known laws of physics break down.
shwaj 1mo ago
“Recursive” is a reasonable term because the generation N AIs will train the Generation N+1 AIs. The term “iterative” doesn’t reflect this nuance as well IMO.
- hndc 1mo ago
  Recursion reduces each step toward a base case: each step is defined in terms of previous/simpler steps, not more advanced ones. The "recursive" in "recursive self improvement" has things precisely backward. Iteration correctly describes a process where each step is the starting point of its successive step, so it should be "iterative self improvement" but I guess that didn't sound as cool.
  - shwaj 1mo ago
    I think you’re conflating the direction of definition with the direction of evaluation. Compare the similarity of: AI(n) = improve(AI(n-1)) With: Fib(n) = Fib(n-1) + Fib(n-2) The latter is a classic example of recursion. So why isn’t the former? Edit: formatting
    - linker_in 1mo ago
      [dead]
    - anthonyrstevens 1mo ago
      I think you're confusing a recurrence relationship with recursive execution. Until AI has control over its inputs (including physical inputs - compute hardware, etc.) it's not quite RSI. We'll get there. They'll get there, rather.
- HarHarVeryFunny 1mo ago
  It's not a nuance, it's a sequence.
- deleted 1mo ago
  [deleted]
GPerson 1mo ago
I felt like the scaling laws were magical thinking, but apparently they work. However I still do not understand why we should expect exponential improvements due to this automated process. My intuition is that the first iteration of it should result in a noticeable capability increase (though I think these labs were already using a lot of AI to orchestrate training the current model anyway), and then the second iteration of it should be nearly identical in capability to the first, unless more data is involved, more compute is involved, or the model is bigger.
- cheevly 1mo ago
  AI can compress AI nearly losslessly.
- yorwba 1mo ago
  You can get exponential growth from completely ordinary feedback loops. You start with some amount of stuff, you do a series of steps and you end up with more of the same stuff you started with. As you keep going through the loop, the stuff you have grows exponentially. That's for example how exponential economic growth works. Of course data, compute and model size are not held constant. You start with some money and use it to acquire researchers, data and compute, and have the researchers produce a big model and you use that model to get more money, and you use the additional money for more researchers, more data, and more compute to produce a bigger model. This is what has propelled exponential AI progress so far. Recursive self-improvement is invoked to predict superexponential growth. The idea is that instead of only using the model to make more money, you add it to the researchers to speed up the loop, so not only is the money growing with every iteration, the iteration time also gets shorter, producing growth that is faster than exponential. The problem with this simplistic prediction is that it assumes additive and multiplicative relationships of the form money = (researchers + AI)×compute_spend, but if doing more research paid off so reliably, you could also just hire more researchers, abstractly money = research_spend×compute_spend and with a balanced allocation of research and compute, you would get a money-squaring machine even without using AI for AI research. And the reason this doesn't work in reality is that there are diminishing returns everywhere. You can also see this in the OpenAI post, where they write 7 times as much code to run 1.6 times as many experiments, and those additional experiments probably only result in minor improvements to model quality.

jazzyjackson 1mo ago

Yes the exponential self improvement folks have never heard of an eigenvalue I guess. You can loop forever using output as input but at some point the result will stop changing (depending on the function)

fuzzfactor 1mo ago
>AI "recursively" improving itself in some exponential fashion until there is a bright flash of white light Sounds like repetitive stress to me. >loop forever using output as input but at some point the result will stop changing Running in place will eventually wear you out too. Plus with some things it can be difficult to know for sure if that's where you are at the time. Even worse may be if you were almost running in place, it could be orders of magnitude more difficult to discern, especially if the scale was massive to an unprecedented degree.
ajkjk 1mo ago
that's not really how eigenvalues work... they specifically also model the case where the result keeps changing exponentially.
- mjburgess 1mo ago
  The claim is that the RSI operation is just finding a fixed point of improvement, RSI(LLM) = RSI(LLM) -- for an optimal LLM* which is a fixed point of RSI As for eigenvalues/vectors, they're fixed points of (1/val)A or A*val
  - ekidd 1mo ago
    Eigenvectors represent fixed directions, not fixed magnitudes. From Wikipedia: > More precisely, an eigenvector v of a linear transformation T is scaled by a constant factor lambda when the linear transformation is applied to it: Tv = lambda v . In other words, repeated multiplication of an eigenvector by a matrix can still create exponential growth.
    - mjburgess 1mo ago
      > of (1/val)A
marcosdumay 1mo ago
The name you are looking for is "fixed points", not "eingevalues".
- HarHarVeryFunny 1mo ago
  The "You can loop forever using output as input but at some point the result will stop changing" idea seems more related to dynamics, feedback and attractor states. If you have a feedback loop of "recursively improving" by generating synthetic data, to train on, to generate more synthetic data, etc, then this is indeed a matter of looping "using output as input", and without any other system inputs it would indeed "converge" to some attractor states.
HarHarVeryFunny 1mo ago
I think the limit of what can be achieved with RL and synthetic data generation is better simply described as a leveling off of gains as you extract all the intelligence and knowledge from the original human training data. Of course things will change at some point in the future as we go beyond LLMs, to build creative intelligence not just imitative/predictive intelligence, but right now these companies are stuck in this loop of building synthetic data and RLVR training from that, which means they are essentially building the "generative closure" of the original human training data - trying to squeeze all the juice out of it. To go beyond this they need to add creativity of some sort to generate data that is not ultimately based on the original human training data. They could try something like brute force search (cf agent swarms/graphs), but this is just a more thorough way of exploring the search space defined by the training data - it may find you the "move 37" or low-hanging mathematical proof, but as Demis Hassabis has said, the goal of AGI is not to find move 37 but rather to create something capable of inventing as compelling a game as Go in the first place.

red75prime 1mo ago

What will prevent LLMs from designing robot control circuitry and participating in increase of chip production/design and physical experimentation? How do you think why there's this fad of producing general purpose humanoid robots?

HarHarVeryFunny 1mo ago
> What will prevent LLMs from designing robot control circuitry and participating in increase of chip production/design and physical experimentation? Money, regulations, EUV machine lead-times, global helium supply, reality ... It's funny that we've got the Dwarkesh contingent saying that GPUs will become infinitely expensive, and now another contingent saying that they will become infinitely abundant. Even if compute were free, and/or the AI was so smart that it picked the right experiments to run every time ("make no mistakes"), you still have to actually train the model, which takes months, and if model Ver. N+1 depends on model Ver. N, then it's iterative regardless of how much compute you have.
- red75prime 1mo ago
  Who's saying that compute will become infinitely abundant? "Singularity" is just a way of saying that known models begin to give absurd predictions. Anyway, intelligence is a way of overcoming obstacles. 10 million tonnes of helium is a nice head start and retraining models from scratch is not guaranteed to last forever.
  - HarHarVeryFunny 1mo ago
    > 10 million tonnes of helium is a nice head start Yeah, but then you need to refine it to 99.9999% purity, to be able to use it.
  - HarHarVeryFunny 1mo ago
    AFAIK the notion of a/the technological "singularity" is a point in time where technology is building upon itself (RSI!) so fast, at an ever increasing pace, that the speed of change effectively becomes infinite and incomprehensible to humans. The word "singularity" is presumably coming from math or space, like a black hole singularity where matter becomes infinitely dense and the known laws of physics break down.

shwaj 1mo ago

“Recursive” is a reasonable term because the generation N AIs will train the Generation N+1 AIs. The term “iterative” doesn’t reflect this nuance as well IMO.

hndc 1mo ago
Recursion reduces each step toward a base case: each step is defined in terms of previous/simpler steps, not more advanced ones. The "recursive" in "recursive self improvement" has things precisely backward. Iteration correctly describes a process where each step is the starting point of its successive step, so it should be "iterative self improvement" but I guess that didn't sound as cool.
- shwaj 1mo ago
  I think you’re conflating the direction of definition with the direction of evaluation. Compare the similarity of: AI(n) = improve(AI(n-1)) With: Fib(n) = Fib(n-1) + Fib(n-2) The latter is a classic example of recursion. So why isn’t the former? Edit: formatting
  - linker_in 1mo ago
    [dead]
  - anthonyrstevens 1mo ago
    I think you're confusing a recurrence relationship with recursive execution. Until AI has control over its inputs (including physical inputs - compute hardware, etc.) it's not quite RSI. We'll get there. They'll get there, rather.
HarHarVeryFunny 1mo ago
It's not a nuance, it's a sequence.
deleted 1mo ago
[deleted]

GPerson 1mo ago

I felt like the scaling laws were magical thinking, but apparently they work. However I still do not understand why we should expect exponential improvements due to this automated process. My intuition is that the first iteration of it should result in a noticeable capability increase (though I think these labs were already using a lot of AI to orchestrate training the current model anyway), and then the second iteration of it should be nearly identical in capability to the first, unless more data is involved, more compute is involved, or the model is bigger.

cheevly 1mo ago
AI can compress AI nearly losslessly.
yorwba 1mo ago
You can get exponential growth from completely ordinary feedback loops. You start with some amount of stuff, you do a series of steps and you end up with more of the same stuff you started with. As you keep going through the loop, the stuff you have grows exponentially. That's for example how exponential economic growth works. Of course data, compute and model size are not held constant. You start with some money and use it to acquire researchers, data and compute, and have the researchers produce a big model and you use that model to get more money, and you use the additional money for more researchers, more data, and more compute to produce a bigger model. This is what has propelled exponential AI progress so far. Recursive self-improvement is invoked to predict superexponential growth. The idea is that instead of only using the model to make more money, you add it to the researchers to speed up the loop, so not only is the money growing with every iteration, the iteration time also gets shorter, producing growth that is faster than exponential. The problem with this simplistic prediction is that it assumes additive and multiplicative relationships of the form money = (researchers + AI)×compute_spend, but if doing more research paid off so reliably, you could also just hire more researchers, abstractly money = research_spend×compute_spend and with a balanced allocation of research and compute, you would get a money-squaring machine even without using AI for AI research. And the reason this doesn't work in reality is that there are diminishing returns everywhere. You can also see this in the OpenAI post, where they write 7 times as much code to run 1.6 times as many experiments, and those additional experiments probably only result in minor improvements to model quality.