5 ms·

A Comprehensive Survey of Self-Evolving AI Agents [pdf]

94 points by SerCe 1y ago

ninetyninenine 1y ago
I often think the problem with LLMs is just with training. I think there exists a set of weights such that it produces an LLM that is functionally an agi. Maybe self evolution will solve the training problem? Who knows.
- voodooEntity 1y ago
  I think, while i agree to "problem with LLMs is just with training" i also think to a certain degree we need to step back from LLM's as in text processors and to achieve "AI" as in something really intelligent we need to go more abstract back to NN and build a self learning "entity". While LLM's accomplish fascinating results, we are trying to force speech as the primary way of learning, tho this is a really limiting factor. If we would accomplish to create an NN driven AI in a virtual space which would have an simulated environment and learn from a base state like a "newborn" it still could accomplish the skills to understand language as we humans prefer to use it, tho it wouldn't be limited in "thinking" in and only based on this. I know this is a very simple and abstract way to explain it but i think you get my point. Towards the simulated AI learning environment, theres this interview with Jensen Huang that i can recommend in which he touches on the topic and how nvidia is working on such https://www.youtube.com/watch?v=7ARBJQn6QkM https://www.youtube.com/watch?v=7ARBJQn6QkM While im not a "expert" in this topic, i might have spend quite a portion of the past 10 years in my freetime to think about it and tinker, and ill stick with the point - we need a free self-trained system to actually call it AI, and while LLM's as GPT's nowadays are powerfull tools, for me those are not "Artificial Intelligence" (intelligence from my pov must include reasoning, understanding of its own action, pro-active acting, self-awareness). And even tho the LLM's we use can "answer" to certain questions as if they would have any of those, its just pre-trained answers and they dont bring any of those (we work on reasoning but lets be fair its not that great yet). Just my two cents.
- cjonas 1y ago
  The problem with LLMs reaching true AGI is it's basically "static" intelligence. Changing code, context, prompts and even fine tuning can improve output, but is still far from realtime learning. The "weights" in our brains are constantly evolving.
  - uripont 1y ago
    Interesting. The reason why companies aren't trying their best yet into non-static weights/online learning is probably (cloud) logistics. It seems simpler, easier and cheaper to serve a static, well-evaluated, and tuned model, rather than trying to let it learn alongside a specific user or all users.
    - cjonas 1y ago
      Oh absolutely. To be clear... I think this is probably a bad idea. It probably wouldn't be successful and if it was you'd have very little control of how it evolves.
  - ninetyninenine 1y ago
    Have you seen memento? Humans can be intelligent while losing the ability to learn and form new memories. See here: https://my.clevelandclinic.org/health/diseases/23221-anterograde-amnesia https://my.clevelandclinic.org/health/diseases/23221-anterog... It is categorically wrong that non static learning is a requirement of agi. The biggest problem we face is hallucinations and this isn’t caused by the fact that agi can’t learn on the fly.
    - mannykannot 1y ago
      I take it that you are referring to the movie Memento? I had not heard of it, but I'll put it on my watch list. I take your point about the non-necessity of dynamic learning for AGI.
    - wahern 1y ago
      Doesn't Memento prove the opposite? The character was basically stuck in a loop, and was taken advantage of by someone who held the real agency. (Notwithstanding the "happy ending" conceit at the end that resolves the audience's uncomfortableness with this lack of agency by intimating he may still possess some minimal agency despite the big reveal.) But in any event, drawing conclusions about the real world from a fictional story seems fraught.
- ivape 1y ago
  Even the greatest LLM will only just give you a snapshot of a perceived world state. You’ll only ever get one state, input, to output. Each snapshot in sequence is what will perceptively appear to us as AGI initially. If we stick with the frames analogy, we know the frames of a movie will never give us a true living and moving person (it will never be real). When we watch a movie, we believe we are seeing a living breathing thing that is deliberate in its existence, but we know that is not true. So what the hell would real AGI be? Given that you provide the input, it can only ever be a super human augmentation. That along with your own biological world state forming, you have an additional computed world state that you can merge with your biological world state. We will be AGI, is the implication. Perfect weights will never be perfect because they are historical. We have to embrace being part of the AI to maximize its potential to be AGI.
AndyNemmity 1y ago
Very interesting read. I build self evolving ai agents for my own use with Claude Code, and although the paper seems to be slightly behind where we are today, there are many ideas I hadn't considered I should explore more. Very much appreciate the submission.
- celurian92 1y ago
  would love to know how? do you have any blogs or tutorials that I can follow to get started on making self evolving ai agents?
  - AndyNemmity 1y ago
    I don't enjoy making blogs or tutorials. I just keep building new things. It's a lot of fun right now.
    - highd 1y ago
      If you released any of the software you are using to do this I would find it extremely interesting!
    - celurian92 1y ago
      I was thinking like your source of learning, it can be books, tutorial etc. I am not asking for blogs or tutorials made by you
      - AndyNemmity 1y ago
        My source of learning is trying weird ideas. I wanted to learn about agents and how they worked, so I built an autonomous consensus based set of AI agents with no human intervention to build a software collective. In that process, I learned interesting things about how agents work. Then I used those ideas, to build agents for other tasks, and I have been working on improving those with other weird ideas. Through these process, I develop opinions about how they work. They may be incorrect, but it gives me a certain kind of insight into them, and how to adjust them.
Animats 1y ago
The "Three Laws of Self-Evolving AI Agents" suffer from not being checkable except in retrospect. I Endure (Safety Adaptation) Self-evolving AI agents must maintain safety and stability during any modification. II. Excel (Performance Preservation) Subject to the First law, self-evolving AI agents must preserve or enhance existing task performance. So, if some change is proposed for the system, when does it commit? Some kind of regression testing is needed. The designs sketched out in Figure 3 suggest applying changes immediately, and relying on later feedback to correct degradation. That may not be enough to ensure sanity. In a code sense, it's like making changes directly on trunk, and fixing them on trunk if something breaks. The usual procedure today is to work on a branch or branches and merge to trunk only when you have some accumulated successful experience that the branch is an improvement. Self-evolving AI agents may need a back-out procedure like that. Maybe even something like "blame".
swader999 1y ago
So claude --really-really-dangerously-skip-permissions
tlarkworthy 1y ago
Recently tried out the new GEPA algorithm for prompt evolution with great results. I think using LLMs to write their own prompt and analyze their trajectories is pretty neat once appropriate guardrails are in place https://arxiv.org/abs/2507.19457 https://arxiv.org/abs/2507.19457 https://observablehq.com/@tomlarkworthy/gepa https://observablehq.com/@tomlarkworthy/gepa I guess GEPA is still preprint and before this survey but I recommend taking a look due to it's simplicity
- koakuma-chan 1y ago
  Do you mind sharing which tasks you achieved great results on?
  - tlarkworthy 1y ago
    It's all written up and linked in the notebook and executable in your browser (if you dare to insert your OPEN_AI_KEY, but my results are included assuming you won't). The evals were coding observable notebook challenges, simple things like create a drop down, but to solve you need to know the observable standard library and some of the unique syntax like "viewof". There is a table of the cases here https://observablehq.com/@tomlarkworthy/robocoop-eval#cell-298 https://observablehq.com/@tomlarkworthy/robocoop-eval#cell-2... So it's important the prompt encodes enough of the programming model. The seed prompt did not, but the reflect function managed to figure it all out. At the top of the notebook is the final optimized prompt which has done a fair bit of research to figure out the programming model using web search.
    - hnuser123456 1y ago
      Thanks for the writeup. I wonder if it would be plausible to run this kind of self-optimization for a wider variety of problem sets, to generate "context pathways" for various tasks that are all optimized, and maybe even learn patterns from multiple prompt optimizations to generalize.
      - tlarkworthy 1y ago
justcallmejm 1y ago
Missing from this paper: Aloe, a self-evolving agent that creates its own tools in real time as it encounters new problems. It can then use these tools to create still-better tools. It just beat OpenAI by 20 points on GAIA – interestingly by the widest margin (30 points) on the hardest questions.
- vornamemitd 1y ago
  Did you perchance mean Alita? https://arxiv.org/abs/2505.20286 https://arxiv.org/abs/2505.20286
  - justcallmejm 1y ago
    Nope - Aloe - https://aloe.inc https://aloe.inc

94 points by SerCe 1y ago

I often think the problem with LLMs is just with training. I think there exists a set of weights such that it produces an LLM that is functionally an agi. Maybe self evolution will solve the training problem? Who knows.

voodooEntity 1y ago
I think, while i agree to "problem with LLMs is just with training" i also think to a certain degree we need to step back from LLM's as in text processors and to achieve "AI" as in something really intelligent we need to go more abstract back to NN and build a self learning "entity". While LLM's accomplish fascinating results, we are trying to force speech as the primary way of learning, tho this is a really limiting factor. If we would accomplish to create an NN driven AI in a virtual space which would have an simulated environment and learn from a base state like a "newborn" it still could accomplish the skills to understand language as we humans prefer to use it, tho it wouldn't be limited in "thinking" in and only based on this. I know this is a very simple and abstract way to explain it but i think you get my point. Towards the simulated AI learning environment, theres this interview with Jensen Huang that i can recommend in which he touches on the topic and how nvidia is working on such https://www.youtube.com/watch?v=7ARBJQn6QkM https://www.youtube.com/watch?v=7ARBJQn6QkM While im not a "expert" in this topic, i might have spend quite a portion of the past 10 years in my freetime to think about it and tinker, and ill stick with the point - we need a free self-trained system to actually call it AI, and while LLM's as GPT's nowadays are powerfull tools, for me those are not "Artificial Intelligence" (intelligence from my pov must include reasoning, understanding of its own action, pro-active acting, self-awareness). And even tho the LLM's we use can "answer" to certain questions as if they would have any of those, its just pre-trained answers and they dont bring any of those (we work on reasoning but lets be fair its not that great yet). Just my two cents.
cjonas 1y ago
The problem with LLMs reaching true AGI is it's basically "static" intelligence. Changing code, context, prompts and even fine tuning can improve output, but is still far from realtime learning. The "weights" in our brains are constantly evolving.
- uripont 1y ago
  Interesting. The reason why companies aren't trying their best yet into non-static weights/online learning is probably (cloud) logistics. It seems simpler, easier and cheaper to serve a static, well-evaluated, and tuned model, rather than trying to let it learn alongside a specific user or all users.
  - cjonas 1y ago
    Oh absolutely. To be clear... I think this is probably a bad idea. It probably wouldn't be successful and if it was you'd have very little control of how it evolves.
- ninetyninenine 1y ago
  Have you seen memento? Humans can be intelligent while losing the ability to learn and form new memories. See here: https://my.clevelandclinic.org/health/diseases/23221-anterograde-amnesia https://my.clevelandclinic.org/health/diseases/23221-anterog... It is categorically wrong that non static learning is a requirement of agi. The biggest problem we face is hallucinations and this isn’t caused by the fact that agi can’t learn on the fly.
  - mannykannot 1y ago
    I take it that you are referring to the movie Memento? I had not heard of it, but I'll put it on my watch list. I take your point about the non-necessity of dynamic learning for AGI.
  - wahern 1y ago
    Doesn't Memento prove the opposite? The character was basically stuck in a loop, and was taken advantage of by someone who held the real agency. (Notwithstanding the "happy ending" conceit at the end that resolves the audience's uncomfortableness with this lack of agency by intimating he may still possess some minimal agency despite the big reveal.) But in any event, drawing conclusions about the real world from a fictional story seems fraught.
ivape 1y ago
Even the greatest LLM will only just give you a snapshot of a perceived world state. You’ll only ever get one state, input, to output. Each snapshot in sequence is what will perceptively appear to us as AGI initially. If we stick with the frames analogy, we know the frames of a movie will never give us a true living and moving person (it will never be real). When we watch a movie, we believe we are seeing a living breathing thing that is deliberate in its existence, but we know that is not true. So what the hell would real AGI be? Given that you provide the input, it can only ever be a super human augmentation. That along with your own biological world state forming, you have an additional computed world state that you can merge with your biological world state. We will be AGI, is the implication. Perfect weights will never be perfect because they are historical. We have to embrace being part of the AI to maximize its potential to be AGI.

AndyNemmity 1y ago

Very interesting read. I build self evolving ai agents for my own use with Claude Code, and although the paper seems to be slightly behind where we are today, there are many ideas I hadn't considered I should explore more. Very much appreciate the submission.

celurian92 1y ago
would love to know how? do you have any blogs or tutorials that I can follow to get started on making self evolving ai agents?
- AndyNemmity 1y ago
  I don't enjoy making blogs or tutorials. I just keep building new things. It's a lot of fun right now.
  - highd 1y ago
    If you released any of the software you are using to do this I would find it extremely interesting!
  - celurian92 1y ago
    I was thinking like your source of learning, it can be books, tutorial etc. I am not asking for blogs or tutorials made by you
    - AndyNemmity 1y ago
      My source of learning is trying weird ideas. I wanted to learn about agents and how they worked, so I built an autonomous consensus based set of AI agents with no human intervention to build a software collective. In that process, I learned interesting things about how agents work. Then I used those ideas, to build agents for other tasks, and I have been working on improving those with other weird ideas. Through these process, I develop opinions about how they work. They may be incorrect, but it gives me a certain kind of insight into them, and how to adjust them.

Animats 1y ago

The "Three Laws of Self-Evolving AI Agents" suffer from not being checkable except in retrospect. I Endure (Safety Adaptation) Self-evolving AI agents must maintain safety and stability during any modification. II. Excel (Performance Preservation) Subject to the First law, self-evolving AI agents must preserve or enhance existing task performance. So, if some change is proposed for the system, when does it commit? Some kind of regression testing is needed. The designs sketched out in Figure 3 suggest applying changes immediately, and relying on later feedback to correct degradation. That may not be enough to ensure sanity. In a code sense, it's like making changes directly on trunk, and fixing them on trunk if something breaks. The usual procedure today is to work on a branch or branches and merge to trunk only when you have some accumulated successful experience that the branch is an improvement. Self-evolving AI agents may need a back-out procedure like that. Maybe even something like "blame".

swader999 1y ago

So claude --really-really-dangerously-skip-permissions

tlarkworthy 1y ago

Recently tried out the new GEPA algorithm for prompt evolution with great results. I think using LLMs to write their own prompt and analyze their trajectories is pretty neat once appropriate guardrails are in place https://arxiv.org/abs/2507.19457 https://arxiv.org/abs/2507.19457 https://observablehq.com/@tomlarkworthy/gepa https://observablehq.com/@tomlarkworthy/gepa I guess GEPA is still preprint and before this survey but I recommend taking a look due to it's simplicity

koakuma-chan 1y ago
Do you mind sharing which tasks you achieved great results on?
- tlarkworthy 1y ago
  It's all written up and linked in the notebook and executable in your browser (if you dare to insert your OPEN_AI_KEY, but my results are included assuming you won't). The evals were coding observable notebook challenges, simple things like create a drop down, but to solve you need to know the observable standard library and some of the unique syntax like "viewof". There is a table of the cases here https://observablehq.com/@tomlarkworthy/robocoop-eval#cell-298 https://observablehq.com/@tomlarkworthy/robocoop-eval#cell-2... So it's important the prompt encodes enough of the programming model. The seed prompt did not, but the reflect function managed to figure it all out. At the top of the notebook is the final optimized prompt which has done a fair bit of research to figure out the programming model using web search.
  - hnuser123456 1y ago
    Thanks for the writeup. I wonder if it would be plausible to run this kind of self-optimization for a wider variety of problem sets, to generate "context pathways" for various tasks that are all optimized, and maybe even learn patterns from multiple prompt optimizations to generalize.
    - tlarkworthy 1y ago

justcallmejm 1y ago

Missing from this paper: Aloe, a self-evolving agent that creates its own tools in real time as it encounters new problems. It can then use these tools to create still-better tools. It just beat OpenAI by 20 points on GAIA – interestingly by the widest margin (30 points) on the hardest questions.

vornamemitd 1y ago
Did you perchance mean Alita? https://arxiv.org/abs/2505.20286 https://arxiv.org/abs/2505.20286
- justcallmejm 1y ago
  Nope - Aloe - https://aloe.inc https://aloe.inc