4 ms·

Claude Fable 5 beats Pokémon FireRed using only vision: https://www.youtube.com/watch?v=CIQBP1w4B1M https://www.youtube.com/watch?v=CIQBP1w4B1M

suddenlybananas 4mo ago
Is there any more detail about this besides the very fast slideshow?
- modeless 4mo ago
  Seems like the harness was minimal with no extra game state or maps available. Apparently just the screen image. Seems like it took 50 hours in game time which according to Google is at the high end of a normal human playthrough. No idea how long it took in real time though.
svcphr 4mo ago
Bold move putting in the lvl 3 Pidgey against Gary's Blastoise at the end there (~14sec in... integer timestamps insufficient here).
ex-aws-dude 4mo ago
I mean that’s AGI confirmed right?
uludag 4mo ago
Any suggestion on how I should calibrate my cynicism towards this? I can immagine Anthropic running this experiment multiple times and picking the most impressive one. Or I could immagine like this entire run costing like $1000+ of tokens for this particular run. Or maybe they tried a bunch of Pokemon games and it couldn't even finish some of them. Or is it just able to do this because it has an immense amount of FireRed training data, and if you were to give it an "original" Pokemon game, where it actually had to navigate novel circumstances it would fail.
- modeless 4mo ago
  Every model has encyclopedic knowledge of Pokémon FireRed, of course. Knowledge is not ability. This is the first model with the ability to apply that knowledge to beat the game without assistance. I highly doubt they focused on FireRed specifically in pretraining or posttraining. But we'll see when the ARC-AGI-3 results come out. That will measure its performance on unseen games. Based on this I expect the ARC-AGI-3 score to be SOTA.
milkkarten 4mo ago
no reasoning shown. no explanation on any training information. Using vision-only should be an easier version of the task (given training). there are many standardized evals to do this correctly and Anthropic ignored them to provide a 18 second sped up video of a 50 hour run? yeah I don't trust this until they provide a live run by a 3rd party with full reasoning traces in real-time. The reason we all liked the Gemini Plays Pokemon style runs were because they were live and couldn't be faked
hmokiguess 4mo ago
"Computer system goes through a finite state machine"
charcircuit 4mo ago
The video is privated now, but the timelapse is weird. Sometimes it skips only seconds before the next screenshot and sometimes it skips probably hours forward.
- vaughands 4mo ago
  It's impressive but it seems like they cut out a ton of the grinding to level up the Charizard to brute force through the game. While this is no small feat, it looks like Claude was unable to actually use a balanced team and instead just circle-square-pegged this thing. :)
xinpw8 4mo ago
hi, pokemon red expert here: that video has since been taken private. there is a new what i would assume to be version of that video posted here https://www.youtube.com/watch?v=Ty_50J84fMY https://www.youtube.com/watch?v=Ty_50J84fMY and heavily redacted with most of the game actually omitted. very possibly this is just another case of anthropic protecting us from their models' immense power
- gusmally 4mo ago
  lmao! thank you for the update; i was wondering why it was no longer viewable.
ml-doom 4mo ago
[dead]

suddenlybananas 4mo ago

Is there any more detail about this besides the very fast slideshow?

modeless 4mo ago
Seems like the harness was minimal with no extra game state or maps available. Apparently just the screen image. Seems like it took 50 hours in game time which according to Google is at the high end of a normal human playthrough. No idea how long it took in real time though.

svcphr 4mo ago

Bold move putting in the lvl 3 Pidgey against Gary's Blastoise at the end there (~14sec in... integer timestamps insufficient here).

ex-aws-dude 4mo ago

I mean that’s AGI confirmed right?

uludag 4mo ago

Any suggestion on how I should calibrate my cynicism towards this? I can immagine Anthropic running this experiment multiple times and picking the most impressive one. Or I could immagine like this entire run costing like $1000+ of tokens for this particular run. Or maybe they tried a bunch of Pokemon games and it couldn't even finish some of them. Or is it just able to do this because it has an immense amount of FireRed training data, and if you were to give it an "original" Pokemon game, where it actually had to navigate novel circumstances it would fail.

modeless 4mo ago
Every model has encyclopedic knowledge of Pokémon FireRed, of course. Knowledge is not ability. This is the first model with the ability to apply that knowledge to beat the game without assistance. I highly doubt they focused on FireRed specifically in pretraining or posttraining. But we'll see when the ARC-AGI-3 results come out. That will measure its performance on unseen games. Based on this I expect the ARC-AGI-3 score to be SOTA.

milkkarten 4mo ago

no reasoning shown. no explanation on any training information. Using vision-only should be an easier version of the task (given training). there are many standardized evals to do this correctly and Anthropic ignored them to provide a 18 second sped up video of a 50 hour run? yeah I don't trust this until they provide a live run by a 3rd party with full reasoning traces in real-time. The reason we all liked the Gemini Plays Pokemon style runs were because they were live and couldn't be faked

hmokiguess 4mo ago

"Computer system goes through a finite state machine"

charcircuit 4mo ago

The video is privated now, but the timelapse is weird. Sometimes it skips only seconds before the next screenshot and sometimes it skips probably hours forward.

vaughands 4mo ago
It's impressive but it seems like they cut out a ton of the grinding to level up the Charizard to brute force through the game. While this is no small feat, it looks like Claude was unable to actually use a balanced team and instead just circle-square-pegged this thing. :)

xinpw8 4mo ago

hi, pokemon red expert here: that video has since been taken private. there is a new what i would assume to be version of that video posted here https://www.youtube.com/watch?v=Ty_50J84fMY https://www.youtube.com/watch?v=Ty_50J84fMY and heavily redacted with most of the game actually omitted. very possibly this is just another case of anthropic protecting us from their models' immense power

gusmally 4mo ago
lmao! thank you for the update; i was wondering why it was no longer viewable.

ml-doom 4mo ago

[dead]