7 ms·

AI agents that “self-reflect” perform better in changing environments

215 points by chdoyle 3y ago

ftxbro 3y ago
So from this hacker news title I definitely thought it was saying that when you give some AI agents a self reflection like maybe by putting an internal monologue loop then they unlock an emergent animal-like exploration behavior. But this is not what happened. Instead, some guys told AI agents to explore in the way that the guys think that animals explore. "Stanford researchers invented the “curious replay” training method based on studying mice to help AI agents"
- onetokeoverthe 3y ago
  [dead]
- sva_ 3y ago
  > Instead, some guys told AI agents to explore in the way that the guys think that animals explore. Something, something, The Bitter Lesson.
- ShamelessC 3y ago
  I hate that titles can differ from the article here. It’s patronizing and commonly inaccurate or misleading.
  - lcnPylGDnU4H9OF 3y ago
    “Patronizing” seems to be a matter of taste. I’ve never considered it to be patronizing; indeed, that’s often much unlike articles which have their title changed. As far as simply differing, much of the time there’s a character limit that’s hit. I’ve seen many posts with comment from the poster calling out their edit to the title and the character limit is usually cited. It would be especially difficult to keep the character limit (I think there are legitimate design reasons for this) while also requiring that the title matches the submission as closely as possible. Who decides what words are omitted without it potentially being any of: patronizing, inaccurate, or misleading?
  - ftxbro 3y ago
    I don't like the misleading titles either, but honestly if you want the real titles you probably want some kind of arxiv feed. The paper title is "Curious Replay for Model-based Adaptation" which is too dry for social media or whatever hacker news is or for whoever is the audience of the stanford university press office. You have to expect more juicy (and therefore somewhat misleading or sensationalized) titles if you don't get your news straight from an arxiv feed.
- baybal2 3y ago
  [dead]
- neuronerd1 3y ago
  Author here, a key thing is that we didn't prescribe that the mechanism of exploration was the same, but rather we found that the AI agent explored poorly (i.e. unlike animals) until we included Curious Replay. Interestingly, we found that the benefits of Curious Replay also led to state of the art performance on Crafter.
  - ftxbro 3y ago
    OK here is the arxiv https://arxiv.org/abs/2306.15934 https://arxiv.org/abs/2306.15934 called "Curious Replay for Model-based Adaptation" and from the abstract it says "we present Curious Replay -- a form of prioritized experience replay tailored to model-based agents through use of a curiosity-based priority signal" and "DreamerV3 with Curious Replay surpasses state-of-the-art performance on Crafter" here is the crafter benchmark https://github.com/danijar/crafter https://github.com/danijar/crafter but it appears to have out of date baselines at the bottom of that page. That arxiv stuff looks perfectly normal but I kind of hate how it got more and more caricatured as it went through the university press office and hacker news clickbait pipeline.
    - tnecniv 3y ago
      That’s standard. Me and others in my PhD cohort have had experiences where we saw so many minor inaccuracies in the copy we only fixed things that were flat out wrong, otherwise we’d have rewritten the whole article. It’s the result of a combination of non-experts having a 30 minute conversation with you then writing based off their notes a week later and the fact that their job is to hype up research so that it gets more attention from a broader audience. Everyone I knew said they wouldn’t let that happen to them when the press office called, but rewriting someone’s whole article because you feel like they missed nuances is hard to take a strong stance on, especially as an early career researcher.
      - ftxbro 3y ago
        yes it's better now that the hn mods have changed the headline
  - 3y ago
- dang 3y ago
  (Submitted title was "“Self-reflecting” AI agents explore like animals". We changed it in keeping with the HN guidelines - https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html.)
  - TechnologyPast 3y ago
    Hi dang. Can you whitelist some URLs for commenting from a new account? Like wikipedia.org and libquotes.com Looks like you shadowbanned this account. Maybe for posting a URL in the first comment.
    - dang 3y ago
      You just got hit by a spam filter. I've turned that off now. But please send questions like this to hn@ycombinator.com, as the site guidelines ask (https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html). They're off topic in the threads.
piyh 3y ago
Direct arxiv link: https://arxiv.org/pdf/2306.15934.pdf https://arxiv.org/pdf/2306.15934.pdf
- sitkack 3y ago
  https://www.semanticscholar.org/paper/Curious-Replay-for-Model-based-Adaptation-Kauvar-Doyle/bd09f1477ff5ea1fe1b9ea57a0272978844ee6ad https://www.semanticscholar.org/paper/Curious-Replay-for-Mod... https://github.com/AutonomousAgentsLab/curiousreplay https://github.com/AutonomousAgentsLab/curiousreplay
FrustratedMonky 3y ago
Exactly. We keep leaving out 'motivation' on these models. Since they are reacting to prompts. But put them on a loop with goals and see what happens. And, things like GPT are not 'embodied', since they don't live in the 'world' they can't associate language with physical reality. Put them in a simulated environment like a game, and it looks a lot more 'conscious'.
xianshou 3y ago
The result is mildly interesting - improvement on an isolated task but none on the full benchmark - but what would be much more compelling is curiosity-driven replay in an LLM context combined with chain- or tree-of-thought techniques. This would be the machine analogy to noticing your confusion, a sort of "what do I need to know" or "what am I overlooking"? Anecdotally, language models perform better when you prompt them to ask their own questions in the process of answering yours, so I would expect curiosity to have a meaningful impact.
jjtheblunt 3y ago
it's kind of interesting how increasingly frequently "stanford.edu" is finding its way into HN submissions, and did the increasing frequency start with the GPT-4 enthusiasm? is that coincidence?
Schnitzkitz 3y ago
Makes sense. AI lacks rationality, and animals lack rationality. Of course, humans are the rational animal, and hence we know when we truly understand things or when we just repeat or spitball.
- mandmandam 3y ago
  Nah, not really. History has been repeating itself for thousands of years. We keep killing the prophets, and putting the absolute worst of us on pedestals. What's rational about that? Dolphins mucking about in the water - that's rational.
  - Schnitzkitz 3y ago
    By pointing to rational or moral failures, you already imply that we are supposed to act in a certain way. If there are people who are the worst, it begs the question of what a good human is, and who or what we should actually follow. Clearly, we don't think that raw power is what makes someone good, because otherwise these worst people on the pedestals would be by default good people, through all the power they have over their followers. If it is irrational that history repeats itself, do you think that it would be rational if history progressed towards some goal, and if yes, what is that goal?
    - mandmandam 3y ago
      > By pointing to rational or moral failures, you already imply that we are supposed to act in a certain way. Don't keep such an open mind that your brain falls out. > If it is irrational that history repeats itself, do you think that it would be rational if history progressed towards some goal It has, often. For example, 50 years ago a bunch of fossil fuel executives decided it would be best to let the planet burn, so they can keep making money. History progressed toward their goal, and now we're starting to really suffer. But they have their megayachts. Do you think that's rational?
      - uoaei 3y ago
riwsky 3y ago
How does this differ from existing approaches that just follow the entropy?
martyvis 3y ago
And here I am halfway through Michael Crichton's novel "Prey" ...
- lannisterstark 3y ago
  Huh. Looks interesting but I have a weird feeling it might be the same old sappy boring thriller. Opinions so far?
  - martyvis 3y ago
    Starting to get interesting. Not sure whether fixing the code, MacGyvering or brute strength will win the day.
ano88888 3y ago
humans need to do self reflection too. It is usually in the form of journaling daily for self reflection
axiom92 3y ago
Some of our recent/relevant work: https://selfrefine.info/ https://selfrefine.info/
ly3xqhl8g9 3y ago
Perhaps one would drop the quotes around self-reflect if one would implement something more akin to a Markov blanket [1], blankets within blankets, model ourselves modelling the world. [1] 2018, "The Markov blankets of life: autonomy, active inference and the free energy principle", https://royalsocietypublishing.org/doi/10.1098/rsif.2017.0792 https://royalsocietypublishing.org/doi/10.1098/rsif.2017.079...
sethammons 3y ago
I'm not an AI expert or even novice nor am I a neuroscientist, but I have been thinking about how I interact with the world. My current imagining says that novelty and unexpected inputs drive our immediate understanding of the world around us. To have expectations you have to have to have a model. When that model breaks and is adjusted you have a novel experience and the model can be updated. This feedback loop is critical. Example: other day I was grilling food and my digital food thermometer was on the metal prep area near the hot griddle. As I was walking away I reached for it, grabbed it, and expected to pick it up. However! I didn't know it had a magnet and it gave me back unexpected stimulus. I immediately jerked my hand away and several thoughts happened near instantly. My thoughts went from I burned my hand to no, no pain, maybe a really bad burn, to no, no heat, no sizzling of flesh, to oops, wrong stimulus, something resisted, resisted how, it slid but wouldn't pick up easy, ah, a magnet. The researchers here are right, I expect. You need curiosity and some goal, but you need to constantly tune the input for expectations and tweak the (mental) model of the world. How many times do you, for a split second, totally misinterpret what you see or feel but near instantly self correct? Better AI will require putting forth it's initial result and then validating the result with feedback. The more unexpected the feedback the more novel the experience and more learning that can happen.

215 points by chdoyle 3y ago

So from this hacker news title I definitely thought it was saying that when you give some AI agents a self reflection like maybe by putting an internal monologue loop then they unlock an emergent animal-like exploration behavior. But this is not what happened. Instead, some guys told AI agents to explore in the way that the guys think that animals explore. "Stanford researchers invented the “curious replay” training method based on studying mice to help AI agents"

onetokeoverthe 3y ago
[dead]
sva_ 3y ago
> Instead, some guys told AI agents to explore in the way that the guys think that animals explore. Something, something, The Bitter Lesson.
ShamelessC 3y ago
I hate that titles can differ from the article here. It’s patronizing and commonly inaccurate or misleading.
- lcnPylGDnU4H9OF 3y ago
  “Patronizing” seems to be a matter of taste. I’ve never considered it to be patronizing; indeed, that’s often much unlike articles which have their title changed. As far as simply differing, much of the time there’s a character limit that’s hit. I’ve seen many posts with comment from the poster calling out their edit to the title and the character limit is usually cited. It would be especially difficult to keep the character limit (I think there are legitimate design reasons for this) while also requiring that the title matches the submission as closely as possible. Who decides what words are omitted without it potentially being any of: patronizing, inaccurate, or misleading?
- ftxbro 3y ago
  I don't like the misleading titles either, but honestly if you want the real titles you probably want some kind of arxiv feed. The paper title is "Curious Replay for Model-based Adaptation" which is too dry for social media or whatever hacker news is or for whoever is the audience of the stanford university press office. You have to expect more juicy (and therefore somewhat misleading or sensationalized) titles if you don't get your news straight from an arxiv feed.
baybal2 3y ago
[dead]
neuronerd1 3y ago
Author here, a key thing is that we didn't prescribe that the mechanism of exploration was the same, but rather we found that the AI agent explored poorly (i.e. unlike animals) until we included Curious Replay. Interestingly, we found that the benefits of Curious Replay also led to state of the art performance on Crafter.
- ftxbro 3y ago
  OK here is the arxiv https://arxiv.org/abs/2306.15934 https://arxiv.org/abs/2306.15934 called "Curious Replay for Model-based Adaptation" and from the abstract it says "we present Curious Replay -- a form of prioritized experience replay tailored to model-based agents through use of a curiosity-based priority signal" and "DreamerV3 with Curious Replay surpasses state-of-the-art performance on Crafter" here is the crafter benchmark https://github.com/danijar/crafter https://github.com/danijar/crafter but it appears to have out of date baselines at the bottom of that page. That arxiv stuff looks perfectly normal but I kind of hate how it got more and more caricatured as it went through the university press office and hacker news clickbait pipeline.
  - tnecniv 3y ago
    That’s standard. Me and others in my PhD cohort have had experiences where we saw so many minor inaccuracies in the copy we only fixed things that were flat out wrong, otherwise we’d have rewritten the whole article. It’s the result of a combination of non-experts having a 30 minute conversation with you then writing based off their notes a week later and the fact that their job is to hype up research so that it gets more attention from a broader audience. Everyone I knew said they wouldn’t let that happen to them when the press office called, but rewriting someone’s whole article because you feel like they missed nuances is hard to take a strong stance on, especially as an early career researcher.
    - ftxbro 3y ago
      yes it's better now that the hn mods have changed the headline
- 3y ago
dang 3y ago
(Submitted title was "“Self-reflecting” AI agents explore like animals". We changed it in keeping with the HN guidelines - https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html.)
- TechnologyPast 3y ago
  Hi dang. Can you whitelist some URLs for commenting from a new account? Like wikipedia.org and libquotes.com Looks like you shadowbanned this account. Maybe for posting a URL in the first comment.
  - dang 3y ago
    You just got hit by a spam filter. I've turned that off now. But please send questions like this to hn@ycombinator.com, as the site guidelines ask (https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html). They're off topic in the threads.

piyh 3y ago

Direct arxiv link: https://arxiv.org/pdf/2306.15934.pdf https://arxiv.org/pdf/2306.15934.pdf

sitkack 3y ago
https://www.semanticscholar.org/paper/Curious-Replay-for-Model-based-Adaptation-Kauvar-Doyle/bd09f1477ff5ea1fe1b9ea57a0272978844ee6ad https://www.semanticscholar.org/paper/Curious-Replay-for-Mod... https://github.com/AutonomousAgentsLab/curiousreplay https://github.com/AutonomousAgentsLab/curiousreplay

FrustratedMonky 3y ago

Exactly. We keep leaving out 'motivation' on these models. Since they are reacting to prompts. But put them on a loop with goals and see what happens. And, things like GPT are not 'embodied', since they don't live in the 'world' they can't associate language with physical reality. Put them in a simulated environment like a game, and it looks a lot more 'conscious'.

xianshou 3y ago

The result is mildly interesting - improvement on an isolated task but none on the full benchmark - but what would be much more compelling is curiosity-driven replay in an LLM context combined with chain- or tree-of-thought techniques. This would be the machine analogy to noticing your confusion, a sort of "what do I need to know" or "what am I overlooking"? Anecdotally, language models perform better when you prompt them to ask their own questions in the process of answering yours, so I would expect curiosity to have a meaningful impact.

jjtheblunt 3y ago

it's kind of interesting how increasingly frequently "stanford.edu" is finding its way into HN submissions, and did the increasing frequency start with the GPT-4 enthusiasm? is that coincidence?

Schnitzkitz 3y ago

Makes sense. AI lacks rationality, and animals lack rationality. Of course, humans are the rational animal, and hence we know when we truly understand things or when we just repeat or spitball.

mandmandam 3y ago
Nah, not really. History has been repeating itself for thousands of years. We keep killing the prophets, and putting the absolute worst of us on pedestals. What's rational about that? Dolphins mucking about in the water - that's rational.
- Schnitzkitz 3y ago
  By pointing to rational or moral failures, you already imply that we are supposed to act in a certain way. If there are people who are the worst, it begs the question of what a good human is, and who or what we should actually follow. Clearly, we don't think that raw power is what makes someone good, because otherwise these worst people on the pedestals would be by default good people, through all the power they have over their followers. If it is irrational that history repeats itself, do you think that it would be rational if history progressed towards some goal, and if yes, what is that goal?
  - mandmandam 3y ago
    > By pointing to rational or moral failures, you already imply that we are supposed to act in a certain way. Don't keep such an open mind that your brain falls out. > If it is irrational that history repeats itself, do you think that it would be rational if history progressed towards some goal It has, often. For example, 50 years ago a bunch of fossil fuel executives decided it would be best to let the planet burn, so they can keep making money. History progressed toward their goal, and now we're starting to really suffer. But they have their megayachts. Do you think that's rational?
    - uoaei 3y ago

riwsky 3y ago

How does this differ from existing approaches that just follow the entropy?

martyvis 3y ago

And here I am halfway through Michael Crichton's novel "Prey" ...

lannisterstark 3y ago
Huh. Looks interesting but I have a weird feeling it might be the same old sappy boring thriller. Opinions so far?
- martyvis 3y ago
  Starting to get interesting. Not sure whether fixing the code, MacGyvering or brute strength will win the day.

ano88888 3y ago

humans need to do self reflection too. It is usually in the form of journaling daily for self reflection

axiom92 3y ago

Some of our recent/relevant work: https://selfrefine.info/ https://selfrefine.info/

ly3xqhl8g9 3y ago

Perhaps one would drop the quotes around self-reflect if one would implement something more akin to a Markov blanket [1], blankets within blankets, model ourselves modelling the world. [1] 2018, "The Markov blankets of life: autonomy, active inference and the free energy principle", https://royalsocietypublishing.org/doi/10.1098/rsif.2017.0792 https://royalsocietypublishing.org/doi/10.1098/rsif.2017.079...

sethammons 3y ago

I'm not an AI expert or even novice nor am I a neuroscientist, but I have been thinking about how I interact with the world. My current imagining says that novelty and unexpected inputs drive our immediate understanding of the world around us. To have expectations you have to have to have a model. When that model breaks and is adjusted you have a novel experience and the model can be updated. This feedback loop is critical. Example: other day I was grilling food and my digital food thermometer was on the metal prep area near the hot griddle. As I was walking away I reached for it, grabbed it, and expected to pick it up. However! I didn't know it had a magnet and it gave me back unexpected stimulus. I immediately jerked my hand away and several thoughts happened near instantly. My thoughts went from I burned my hand to no, no pain, maybe a really bad burn, to no, no heat, no sizzling of flesh, to oops, wrong stimulus, something resisted, resisted how, it slid but wouldn't pick up easy, ah, a magnet. The researchers here are right, I expect. You need curiosity and some goal, but you need to constantly tune the input for expectations and tweak the (mental) model of the world. How many times do you, for a split second, totally misinterpret what you see or feel but near instantly self correct? Better AI will require putting forth it's initial result and then validating the result with feedback. The more unexpected the feedback the more novel the experience and more learning that can happen.