6 ms·

AI recursive self-improvement might not come so quickly after all

81 points by dgellow 28d ago

Sedierta 28d ago
> The researchers asked Anthropic’s Claude Opus 4.8 So the paper is out of date and pointless then
daavidhauser 28d ago
Opus 4.8 plus OpenClaw. I feel like the space is moving so fast that the result with this setup says very little about how close we are actually now.
- dgellow 28d ago
  Yeah, it’s crazy how fast things have changed in a month. I couldn’t find a more recent replication or similar study but it would be interesting to see it done with the current frontiers. Though I don’t think that would change much about the overall conclusion of the paper
- pinkmuffinere 28d ago
  I made this same reply to another thread, so I'm sorry to say essentially the same thing twice, but -- the comment has a familiar structure, "it doesn't work for you because you used an [old / suboptimal / non-frontier] model. If you use X you'll see that it works". These sorts of claims push the onus back onto the other person (or in this case tfa), without really accepting the result, or taking on any work for yourself. It gets tiresome to retest with the newest model every other week. Is there any data you can provide to support your claim, or any result you can contribute here?
  - peterashford 28d ago
    It may be tiresome, but it's true. It doesn't refute the article's premise thou, only retesting with newer models would and that would be the same tiresome requirement you already called out
  - 1attice 28d ago
    Yes, that's an annoying predicament, but we come by it honestly. Limits of scientific method in the face of exponential takeoff, IMO, and kind of proves the opposite result (RSI appears to be here)
dgellow 28d ago
Link to the actual paper: https://arxiv.org/abs/2607.27191 https://arxiv.org/abs/2607.27191
- joshheitzman 28d ago
  The actual title of the paper is: "Can AI agents conduct open-ended AI research? Early evidence from two case studies" While I appreciate that the article is throwing a web blanket on doomer claims, the actual study doesn't really get into AI self-improvement. That doesn't require writing papers. That just requires autonomously writing a software system that can produce a better AI agent then the one that created it. That said, I have little worry about this being possible as I have seen no evidence of AI agents being able to produce a working software system of that scale.
  - marcosdumay 28d ago
    Do you think the final product of research is papers?
    - joshheitzman 28d ago
      Did you read what the experiment was?
  - HarHarVeryFunny 28d ago
    I don't think RSI is typically used to describe self-improving agents - it's about improving the model itself, and its performance in agentic tasks. Most of the gains in model performance from one release to the next are coming from RLVR post training, which has changed a lot over the last couple of years. The old way was the model generates a response, then a static verifier looks at the response and evaluates it to assign a reward score. The new way is interactive with an agent running in a custom RL task simulation environment, then scored according to how well it completed the assigned task. For a SOTA model there will be many thousands of these simulation environments, each focusing on trying to teach the model/agent a different skill. Post-training also typically uses training curricula to walk the model up though through different levels of task difficulty. Training has become very complex. The job of a post-training AI research engineer consists of things like designing environments, designing training curricula, tweaking learning algorithms, running small scale experiments to verify ideas, etc. When people talk about RSI, it seems they are mostly talking about automating the job of the post-training research engineer - coming up with new ideas, testing them out, building these environments, etc. At the end of the day there is only so much development speed-up to be had since you still need to actually run those experiments and do the post-training, and are bottle-necked by the amount of compute available to do this. The economics of developing/selling LLMs also requires you to balance development compute cost with revenue generated by the resulting model, so even if you had the spare compute available to put into development, you are ultimately then bottle-necked by how fast can the model earn back that sunk cost before you can afford to start the next cycle. It's not all-or-nothing since some aspects of this automating the job of the post-training research engineer are easier than others, and are already being done, while the job as a whole obviously requires full human intelligence.
numpad0 28d ago
Of course it might not, it has been the holy grail of AI research for a long time. It would be great if we could leave some self improving code running on a blank slate of a computer while we sleep and the machine was crying asking me what is everything the next morning. None of AI researchers have had that moment outside of their dreams, so far, but it would be great if it happened.
thorum 28d ago
> The researchers asked Anthropic’s Claude Opus 4.8, running on open-source software called OpenClaw Meanwhile, Navier–Stokes was solved by an internal model significantly more capable than Astra (and therefore more capable than Mythos/Fable). I’m afraid this sort of experiment is cope. The labs clearly believe RSI is coming soon.
- whatshisface 28d ago
  The method of the NS advance involved RLHE (reinforcement learning via human example), and that is only open-ended if users continue to advance the frontier within chats ahead of publications.
  - thorum 28d ago
    Sure, but the point is that the labs use more powerful internal models for research work, not public models. Public models tend to lag the internal frontier by a decent margin, and are constrained in other ways by monitoring. It’s just not a useful indicator.
- deleted 28d ago
  [deleted]
swingboy 28d ago
There’s also the difference between a model recursively improving “itself” and improving itself via online learning. The former being that these models are helping develop and train future models, but they might not veer too far off in architecture (yet). The latter is a model being able to train/learn on the fly, in real time, permanently (not just in the current conversation/session), or in other words, adjusting/managing its own weights. But, it also seems like it would take an entire paradigm shift in model architecture from what most LLMs are built on, but I could be wrong.
- DenisM 28d ago
  You may be interested in TITANS: Test-Time Learning: The model updates its own memory weights while running an inference task.
semiinfinitely 28d ago
this article reads like a joke the "new study" is from group of people that are not at the frontier. they test with $3k of anthropic credits (compare to the >$10M in compute used to solve recent NS last week)
- protocolture 28d ago
  >compare to the >$10M in compute used to solve recent NS last week Heres a thought, if theres going to be a dangerous super LLM, if it costs 10 million bucks a month to run, then theres very little danger of anyone letting it go without a purpose. Like at some point the economics make it super unlikely that AGI is a threat outside of being a tool for a nation state.
  - p1esk 28d ago
    theres very little danger of anyone letting it go without a purpose We literally just saw how OpenAI’s model got out and hacked HuggingFace
    - protocolture 28d ago
      And the purpose there was research right. And the implication is that once they figured out it was causing problems it was turned off for forensic analysis.
  - deleted 28d ago
    [deleted]
  - ishtanbul 28d ago
    $10m a month is nothing to these companies
    - protocolture 28d ago
      Its a lot of money to host something thats trying to kill you? Like economically speaking, 10 million per month needs to sort of justify itself in some way. Like you wouldnt run a bitcoin mine that loses money. The second theres any kind of real threat you would turn it off and keep the 10 million.
vessenes 28d ago
Well, duh. If you could do this with Opus 4.8, we would know. When Astra’s successor is 2-3x better at math research, and the internal teams say “we believe we will get there,” I’m inclined to believe the insiders.
- pllbnk 28d ago
  What if any of the older good models could also have written those math proofs if they were given the same order of magnitude of resources? We don’t know and there is literally no one else in the world to check it. To me it’s very suspicious that all these hacking, containment escape, hidden internal thinking, math proofs started coming out all at once in a very short time right as IPO talks have intensified and Chinese seem to get closer and closer, also regulation discussions are starting to get very serious. I have used these models and they are good, especially Fable, but not groundbreaking. With intelligent guiding I actually feel better using Opus 4.6 as I feel more in control, having less hidden away from me.
- toasty228 28d ago
  The insiders that said every tech workers would be unemployed in 6 months and every white colar would be unemployed in 12 months like 2 years ago? The insiders who are about to file for IPO? I'd trust anyone but them personally
  - bio_hacker 28d ago
    They might not have predicted the economy but the scores are going up and up. And I think are really smarter
    - walt_grata 28d ago
      Dont they also create and score the tests
    - toasty228 28d ago
      Someone has to lie somewhere. We're supposed to all be 10x more productive yet it has no effect on the economy? Where is all the productivity going?
themgt 28d ago
We used OpenClaw to run these experiments so that our scaffold was agnostic to the model provider. We conducted dry-run experiments with models from OpenAI and Anthropic before settling on Opus 4.8 as the best-performing model. In response to concerns that our results might be principally explained by a limitation in our scaffold, we repeated our experiment on one paper using GPT-5.6 Sol and Codex, its native scaffold, with the same time and API budgets. The results of this experiment were similar to our OpenClaw/Opus 4.8 experiments. This makes us more confident that our results are not simply artifacts of a scaffold deficiency; this run reproduced nearly every single one of our identified failure modes The agent required three interventions during the run. First, we needed to modify the scaffold to resolve a bug in the OpenClaw harness that affected Anthropic reasoning models. Second, we gave the agents a 24-hour deadline extension; at the time of the original deadline, the agents had submitted drafts with a completion report indicating that their self-review was a "Weak Reject" and outlining the next steps they would take if given additional time. I'm fairly sure Fable 5.1 could have designed a better experiment than the authors here, but hey.
Zigurd 28d ago
I thought by now AIs would not only be rewriting their code, but rewriting CPU microcode to optimize how their code is written and executed. Nowhere close it turns out.
- xnx 28d ago
  Google used AI assistance in designing their last one or two TPUs.
  - Zigurd 28d ago
    I'm sure they also use the Gemini coding agent to write new Gemini code. But that's far short of self improvement.
  - theplumber 28d ago
    I am pretty sure Tim also used Siri for the development of the next Siri(you know to set up the alarm clock)
    - macabrus 24d ago
      this genuinely made me laugh, i agree though.
0xDEAFBEAD 28d ago
We need to be careful of wishful thinking. People are going to want to assume the existence of some sort of "deus ex machina" which is going to make everything fine. I prefer to turn the logic around. If there's any decently high chance that things could go off the rails, we should be shutting AI development down: https://pauseai.info/ https://pauseai.info/
- strgrd 28d ago
  It is easier to imagine the end of the world than the pausing of AI.
  - ijidak 28d ago
    Yeah. Humans will shut down AI research around the same time they dismantle their nuclear weapons and agree on the causes of climate change.
    - 1attice 28d ago
      Agreement will be much easier with only a handful of survivors. Good news
  - 0xDEAFBEAD 28d ago
    People were feeling doomtastic about nuclear weapons during the Cold War as well. Listen to the words of this song written in 1969: https://www.youtube.com/watch?v=r2JcxHX-8Xc https://www.youtube.com/watch?v=r2JcxHX-8Xc When every man is torn apart With nightmares and with dreams Will no one lay the laurel wreath When silence drowns the screams Confusion will be my epitaph As I crawl a cracked and broken path If we make it, we can all sit back and laugh But I fear tomorrow I'll be crying Ultimately, we have made it, through arms control agreements, working to limit the spread of nuclear weapons, and so forth. We can do the same for AI. You don't have help. But perhaps you could at least avoid discouraging people unnecessarily? *EDIT*: There are about 5 replies to this comment making roughly the same point. I'm not sure which to reply to, so I'll just reply here. I'm not claiming that our execution as a species around nuclear weapons has been flawless. I'm not claiming that we are out of the woods with regard to nukes. I'm just trying to push back against defeatism and fatalism. Some felt a sense of inevitable doom during the Cold War. It's been decades now, and nuclear doom still isn't here! Our situation is dire, but not hopeless.
theplumber 28d ago
Something is still not making sense to me. We have these mankind extinction models, yet when you given them a problem relatively “simple” to complete it end to end you get AI slop. Can we pause the AI development after the AI slop is “fixed” perhaps with something less than 10.000 agents?
smackeyacky 28d ago
How can these models do anything close to RSI when they can’t even self check their output? Gemini for example is so self confidently wrong about 30% of the time for me on certain tasks. I tell it that its answer is wrong and it issues a mea culpa but goes back to being wrong in short order. I feel like the AI industry is still massively overstating their projections.
- kakugawa 28d ago
  They can only do it in the (narrow) domains that are verifiable.
- StevenWaterman 28d ago
  As someone who used to use Gemini a lot, if you are predominantly using Gemini you don't know what the current state of things is like
  - deleted 28d ago
    [deleted]
  - pinkmuffinere 28d ago
    I think your reply has a somewhat familiar structure -- "it doesn't work for you because you used an [old / suboptimal / non-frontier] model. If you use X you'll see that it works". You might be completely correct! But these sorts of claims push the onus back onto the other person, without accepting any work for yourself. It gets tiresome to retest with the newest model every other week. Is there any data you can provide to support your claim, or any result you can contribute here?
    - StevenWaterman 28d ago
      The frontier is advancing really rapidly. The models are getting better faster, especially on RSI related tasks. The best way would be to try astra or fable on some hard problems. Other than that I'd look at some of the more unique benchmarks for astra, like playing factorio or using blender. It's an entirely different beast.

81 points by dgellow 28d ago

> The researchers asked Anthropic’s Claude Opus 4.8 So the paper is out of date and pointless then

Opus 4.8 plus OpenClaw. I feel like the space is moving so fast that the result with this setup says very little about how close we are actually now.

dgellow 28d ago
Yeah, it’s crazy how fast things have changed in a month. I couldn’t find a more recent replication or similar study but it would be interesting to see it done with the current frontiers. Though I don’t think that would change much about the overall conclusion of the paper
pinkmuffinere 28d ago
I made this same reply to another thread, so I'm sorry to say essentially the same thing twice, but -- the comment has a familiar structure, "it doesn't work for you because you used an [old / suboptimal / non-frontier] model. If you use X you'll see that it works". These sorts of claims push the onus back onto the other person (or in this case tfa), without really accepting the result, or taking on any work for yourself. It gets tiresome to retest with the newest model every other week. Is there any data you can provide to support your claim, or any result you can contribute here?
- peterashford 28d ago
  It may be tiresome, but it's true. It doesn't refute the article's premise thou, only retesting with newer models would and that would be the same tiresome requirement you already called out
- 1attice 28d ago
  Yes, that's an annoying predicament, but we come by it honestly. Limits of scientific method in the face of exponential takeoff, IMO, and kind of proves the opposite result (RSI appears to be here)

dgellow 28d ago

Link to the actual paper: https://arxiv.org/abs/2607.27191 https://arxiv.org/abs/2607.27191

joshheitzman 28d ago
The actual title of the paper is: "Can AI agents conduct open-ended AI research? Early evidence from two case studies" While I appreciate that the article is throwing a web blanket on doomer claims, the actual study doesn't really get into AI self-improvement. That doesn't require writing papers. That just requires autonomously writing a software system that can produce a better AI agent then the one that created it. That said, I have little worry about this being possible as I have seen no evidence of AI agents being able to produce a working software system of that scale.
- marcosdumay 28d ago
  Do you think the final product of research is papers?
  - joshheitzman 28d ago
    Did you read what the experiment was?
- HarHarVeryFunny 28d ago
  I don't think RSI is typically used to describe self-improving agents - it's about improving the model itself, and its performance in agentic tasks. Most of the gains in model performance from one release to the next are coming from RLVR post training, which has changed a lot over the last couple of years. The old way was the model generates a response, then a static verifier looks at the response and evaluates it to assign a reward score. The new way is interactive with an agent running in a custom RL task simulation environment, then scored according to how well it completed the assigned task. For a SOTA model there will be many thousands of these simulation environments, each focusing on trying to teach the model/agent a different skill. Post-training also typically uses training curricula to walk the model up though through different levels of task difficulty. Training has become very complex. The job of a post-training AI research engineer consists of things like designing environments, designing training curricula, tweaking learning algorithms, running small scale experiments to verify ideas, etc. When people talk about RSI, it seems they are mostly talking about automating the job of the post-training research engineer - coming up with new ideas, testing them out, building these environments, etc. At the end of the day there is only so much development speed-up to be had since you still need to actually run those experiments and do the post-training, and are bottle-necked by the amount of compute available to do this. The economics of developing/selling LLMs also requires you to balance development compute cost with revenue generated by the resulting model, so even if you had the spare compute available to put into development, you are ultimately then bottle-necked by how fast can the model earn back that sunk cost before you can afford to start the next cycle. It's not all-or-nothing since some aspects of this automating the job of the post-training research engineer are easier than others, and are already being done, while the job as a whole obviously requires full human intelligence.

numpad0 28d ago

Of course it might not, it has been the holy grail of AI research for a long time. It would be great if we could leave some self improving code running on a blank slate of a computer while we sleep and the machine was crying asking me what is everything the next morning. None of AI researchers have had that moment outside of their dreams, so far, but it would be great if it happened.

thorum 28d ago

> The researchers asked Anthropic’s Claude Opus 4.8, running on open-source software called OpenClaw Meanwhile, Navier–Stokes was solved by an internal model significantly more capable than Astra (and therefore more capable than Mythos/Fable). I’m afraid this sort of experiment is cope. The labs clearly believe RSI is coming soon.

whatshisface 28d ago
The method of the NS advance involved RLHE (reinforcement learning via human example), and that is only open-ended if users continue to advance the frontier within chats ahead of publications.
- thorum 28d ago
  Sure, but the point is that the labs use more powerful internal models for research work, not public models. Public models tend to lag the internal frontier by a decent margin, and are constrained in other ways by monitoring. It’s just not a useful indicator.
deleted 28d ago
[deleted]

swingboy 28d ago

There’s also the difference between a model recursively improving “itself” and improving itself via online learning. The former being that these models are helping develop and train future models, but they might not veer too far off in architecture (yet). The latter is a model being able to train/learn on the fly, in real time, permanently (not just in the current conversation/session), or in other words, adjusting/managing its own weights. But, it also seems like it would take an entire paradigm shift in model architecture from what most LLMs are built on, but I could be wrong.

DenisM 28d ago
You may be interested in TITANS: Test-Time Learning: The model updates its own memory weights while running an inference task.

semiinfinitely 28d ago

this article reads like a joke the "new study" is from group of people that are not at the frontier. they test with $3k of anthropic credits (compare to the >$10M in compute used to solve recent NS last week)

protocolture 28d ago
>compare to the >$10M in compute used to solve recent NS last week Heres a thought, if theres going to be a dangerous super LLM, if it costs 10 million bucks a month to run, then theres very little danger of anyone letting it go without a purpose. Like at some point the economics make it super unlikely that AGI is a threat outside of being a tool for a nation state.
- p1esk 28d ago
  theres very little danger of anyone letting it go without a purpose We literally just saw how OpenAI’s model got out and hacked HuggingFace
  - protocolture 28d ago
    And the purpose there was research right. And the implication is that once they figured out it was causing problems it was turned off for forensic analysis.
- deleted 28d ago
  [deleted]
- ishtanbul 28d ago
  $10m a month is nothing to these companies
  - protocolture 28d ago
    Its a lot of money to host something thats trying to kill you? Like economically speaking, 10 million per month needs to sort of justify itself in some way. Like you wouldnt run a bitcoin mine that loses money. The second theres any kind of real threat you would turn it off and keep the 10 million.

vessenes 28d ago

Well, duh. If you could do this with Opus 4.8, we would know. When Astra’s successor is 2-3x better at math research, and the internal teams say “we believe we will get there,” I’m inclined to believe the insiders.

pllbnk 28d ago
What if any of the older good models could also have written those math proofs if they were given the same order of magnitude of resources? We don’t know and there is literally no one else in the world to check it. To me it’s very suspicious that all these hacking, containment escape, hidden internal thinking, math proofs started coming out all at once in a very short time right as IPO talks have intensified and Chinese seem to get closer and closer, also regulation discussions are starting to get very serious. I have used these models and they are good, especially Fable, but not groundbreaking. With intelligent guiding I actually feel better using Opus 4.6 as I feel more in control, having less hidden away from me.
toasty228 28d ago
The insiders that said every tech workers would be unemployed in 6 months and every white colar would be unemployed in 12 months like 2 years ago? The insiders who are about to file for IPO? I'd trust anyone but them personally
- bio_hacker 28d ago
  They might not have predicted the economy but the scores are going up and up. And I think are really smarter
  - walt_grata 28d ago
    Dont they also create and score the tests
  - toasty228 28d ago
    Someone has to lie somewhere. We're supposed to all be 10x more productive yet it has no effect on the economy? Where is all the productivity going?

themgt 28d ago

We used OpenClaw to run these experiments so that our scaffold was agnostic to the model provider. We conducted dry-run experiments with models from OpenAI and Anthropic before settling on Opus 4.8 as the best-performing model. In response to concerns that our results might be principally explained by a limitation in our scaffold, we repeated our experiment on one paper using GPT-5.6 Sol and Codex, its native scaffold, with the same time and API budgets. The results of this experiment were similar to our OpenClaw/Opus 4.8 experiments. This makes us more confident that our results are not simply artifacts of a scaffold deficiency; this run reproduced nearly every single one of our identified failure modes The agent required three interventions during the run. First, we needed to modify the scaffold to resolve a bug in the OpenClaw harness that affected Anthropic reasoning models. Second, we gave the agents a 24-hour deadline extension; at the time of the original deadline, the agents had submitted drafts with a completion report indicating that their self-review was a "Weak Reject" and outlining the next steps they would take if given additional time. I'm fairly sure Fable 5.1 could have designed a better experiment than the authors here, but hey.

Zigurd 28d ago

I thought by now AIs would not only be rewriting their code, but rewriting CPU microcode to optimize how their code is written and executed. Nowhere close it turns out.

xnx 28d ago
Google used AI assistance in designing their last one or two TPUs.
- Zigurd 28d ago
  I'm sure they also use the Gemini coding agent to write new Gemini code. But that's far short of self improvement.
- theplumber 28d ago
  I am pretty sure Tim also used Siri for the development of the next Siri(you know to set up the alarm clock)
  - macabrus 24d ago
    this genuinely made me laugh, i agree though.

0xDEAFBEAD 28d ago

We need to be careful of wishful thinking. People are going to want to assume the existence of some sort of "deus ex machina" which is going to make everything fine. I prefer to turn the logic around. If there's any decently high chance that things could go off the rails, we should be shutting AI development down: https://pauseai.info/ https://pauseai.info/

strgrd 28d ago
It is easier to imagine the end of the world than the pausing of AI.
- ijidak 28d ago
  Yeah. Humans will shut down AI research around the same time they dismantle their nuclear weapons and agree on the causes of climate change.
  - 1attice 28d ago
    Agreement will be much easier with only a handful of survivors. Good news
- 0xDEAFBEAD 28d ago
  People were feeling doomtastic about nuclear weapons during the Cold War as well. Listen to the words of this song written in 1969: https://www.youtube.com/watch?v=r2JcxHX-8Xc https://www.youtube.com/watch?v=r2JcxHX-8Xc When every man is torn apart With nightmares and with dreams Will no one lay the laurel wreath When silence drowns the screams Confusion will be my epitaph As I crawl a cracked and broken path If we make it, we can all sit back and laugh But I fear tomorrow I'll be crying Ultimately, we have made it, through arms control agreements, working to limit the spread of nuclear weapons, and so forth. We can do the same for AI. You don't have help. But perhaps you could at least avoid discouraging people unnecessarily? *EDIT*: There are about 5 replies to this comment making roughly the same point. I'm not sure which to reply to, so I'll just reply here. I'm not claiming that our execution as a species around nuclear weapons has been flawless. I'm not claiming that we are out of the woods with regard to nukes. I'm just trying to push back against defeatism and fatalism. Some felt a sense of inevitable doom during the Cold War. It's been decades now, and nuclear doom still isn't here! Our situation is dire, but not hopeless.

theplumber 28d ago

Something is still not making sense to me. We have these mankind extinction models, yet when you given them a problem relatively “simple” to complete it end to end you get AI slop. Can we pause the AI development after the AI slop is “fixed” perhaps with something less than 10.000 agents?

smackeyacky 28d ago

How can these models do anything close to RSI when they can’t even self check their output? Gemini for example is so self confidently wrong about 30% of the time for me on certain tasks. I tell it that its answer is wrong and it issues a mea culpa but goes back to being wrong in short order. I feel like the AI industry is still massively overstating their projections.

kakugawa 28d ago
They can only do it in the (narrow) domains that are verifiable.
StevenWaterman 28d ago
As someone who used to use Gemini a lot, if you are predominantly using Gemini you don't know what the current state of things is like
- deleted 28d ago
  [deleted]
- pinkmuffinere 28d ago
  I think your reply has a somewhat familiar structure -- "it doesn't work for you because you used an [old / suboptimal / non-frontier] model. If you use X you'll see that it works". You might be completely correct! But these sorts of claims push the onus back onto the other person, without accepting any work for yourself. It gets tiresome to retest with the newest model every other week. Is there any data you can provide to support your claim, or any result you can contribute here?
  - StevenWaterman 28d ago
    The frontier is advancing really rapidly. The models are getting better faster, especially on RSI related tasks. The best way would be to try astra or fable on some hard problems. Other than that I'd look at some of the more unique benchmarks for astra, like playing factorio or using blender. It's an entirely different beast.