6 ms·

Sooo... it can play Pokemon. Feels like they had to throw that in after Google IO yesterday. But the real question is now can it beat the game including the Eli

Sooo... it can play Pokemon. Feels like they had to throw that in after Google IO yesterday. But the real question is now can it beat the game including the Elite Four and the Champion. That was pretty impressive for the new Gemini model.

throwaway314155 1y ago
Gemini can beat the game?
- mxwsn 1y ago
  Gemini has beat it already, but using a different and notably more helpful harness. The creator has said they think harness design is the most important factor right now, and that the results don't mean much for comparing Claude to Gemini.
  - throwaway314155 1y ago
    Way offtopic to TFA now, but isn't using an improved harness a bit like saying "I'm going to hardcore as many priors as possible into this thing so it succeeds regardless of its ability to strategize, plan and execute?
    - samrus 1y ago
      it is. the benchmark was somewhat cheated, from the perspective of finding out how the model adjusts and plans within a dynamic reactive environment
    - 11101010001100 1y ago
      They asked gemini to come up with another word for cheating and it came up with 'harness'.
    - silvr 1y ago
      While true to a degree, I think this is largely wrong. Wouldn't it still count as a "harness" if we provided these LLMs with full robotic control of two humanoid arms, so that it could hold a Gameboy and play the game that way? I don't think the lack of that level of human-ness takes away from the demonstration of long-context reasoning that the GPP stream showed. Claude got stuck reasoning its way through one of the more complex puzzle areas. Gemini took a while on it also, but made it through. I don't that difference can be fully attributed up to the harnesses. Obviously, the best thing to do would be to run a SxS in the same harness of the two models. Maybe that will happen?
      - throwaway314155 1y ago
        I can appreciate that the model is likely still highly capable with a good harness. Still, I think this is more in line with ideas from say, speed running (or hell even reinforcement learning) where you want to prove something profound is possible and to do so before others do, you need to accumulate a series of "tricks" (refining exploits/hacking rewards) in order to achieve the goal. but if you use too many tricks you're no longer proving something as profound as originally claimed. In speed running this tends to splinter into multiple categories. Basically, the gane being conpleted by gemini was in an inferior category (however minuscule) of experiment. I get it though. People demanded these types of changes in the CPP twitch chat, because the pain of watching the model fail in slow motion is simply too much.
- klohto 1y ago
  2 weeks ago
minimaxir 1y ago
That Google IO slide was somewhat misleading as the maintainer of Gemini Plays Pokemon had a much better agentic harness that was constantly iterated upon throughout the runtime (e.g. the maintainer had to give specific instructions on how to use Strength to get past Victory Road), unlike Claude Plays Pokemon. The Elite Four/Champion was a non-issue in comparison especially when you have a lv. 81 Blastoise.
- fourier456 1y ago
  Okay, wait though like I want to know the full transcript because that actually is a better / softer benchmark if you measure in terms of the necessary human input.
hansmayer 1y ago
Right, but on the other hand... how is it even useful? Let's say it can beat the game, so what? So it can (kind of) summarise or write my emails - which is something I neither want nor need, they produce mountains of sloppy code, which I would have to end up fixing, and finally they can play a game? Where is the killer app? The gaming approach was exactly the premise of the original AI efforts in the 1960s, that teaching computers to play chess and other 'brainy' games will somehow lead to development of real AI. It ended as we know in the AI nuclear winter.
- minimaxir 1y ago
  It's a fun benchmark, like simonw's pelican riding a bike. Sometimes fun is the best metric.
- samrus 1y ago
  from a foundational research perspective, the pokemon benchmark is one of the most important ones. these models are trained on a static task, text generation, which is to say the state they are operating in does not change as they operate. but now that they are out we are implicitly demanding they do dynamic tasks like coding, navigation, operating in a market, or playing games. this are tasks where your state changes as you operate an example would be that as these models predict the next word, the ground truth of any further words doesnt change. if it misinterprets the word bank in the sentence "i went to the bank" as a river bank rather than a financial bank, the later ground truth wont change, if it was talking about the visit to the financial bank before, it will still be talking about that regardless of the model's misinterpretation. But if a model takes a wrong turn on the road, or makes a weird buy in the stock market, the environment will react and change and suddenly, what it should have done as the n+1th move before isnt the right move anymore, it needs to figure out a route of the freeway first, or deal with the FOMO bullrush it caused by mistakenly buying alot of stock we need to push against these limits to set the stage for the next evolution of AI, RL based models that are trained in dynamic reactive environments in the first place
  - hansmayer 1y ago
    Honestly I have no idea what is this supposed to mean, and the high verbosity of whatever it is trying to prove is not helping it. To repeat: We already tried making computers play games. Ever heard of Deep Blue, and ever heard of it again since the early 2000s?
archon1410 1y ago
Claude Plays Pokemon was the original concept and inspiration behind "Gemini Plays Pokemon". Gemini arguably only did better because it had access to a much better agent harness and was being actively developed during the run. See: https://www.lesswrong.com/posts/7mqp8uRnnPdbBzJZE/is-gemini-now-better-than-claude-at-pokemon https://www.lesswrong.com/posts/7mqp8uRnnPdbBzJZE/is-gemini-...
- montebicyclelo 1y ago
  Not sure "original concept" is quite right, given it had been tried earlier, e.g. here's a 2023 attempt to get gpt-4-vision to play pokemon, (it didn't really work, but it's clearly "the concept") https://x.com/sidradcliffe/status/1722355983643525427 https://x.com/sidradcliffe/status/1722355983643525427
  - archon1410 1y ago
    I see, I wasn't aware of that. The earliest attempt I knew of was from May 2024,[1] while this gpt-4-vision attempt is from November 2023. I guess Claude Plays Pokemon was the first attempt that had any real success (won a badge), and got a lot of attention over its entertaining "chain-of-thought". [1] https://community.aws/content/2gbBSofaMK7IDUev2wcUbqQXTK6/can-claude-play-pokemon-kind-of https://community.aws/content/2gbBSofaMK7IDUev2wcUbqQXTK6/ca...
- silvr 1y ago
  I disagree - this is all an homage to Twitch Plays Pokemon, which was a noteworthy moment in internet culture/history. https://en.wikipedia.org/wiki/Twitch_Plays_Pok%C3%A9mon https://en.wikipedia.org/wiki/Twitch_Plays_Pok%C3%A9mon

throwaway314155 1y ago

Gemini can beat the game?

mxwsn 1y ago
Gemini has beat it already, but using a different and notably more helpful harness. The creator has said they think harness design is the most important factor right now, and that the results don't mean much for comparing Claude to Gemini.
- throwaway314155 1y ago
  Way offtopic to TFA now, but isn't using an improved harness a bit like saying "I'm going to hardcore as many priors as possible into this thing so it succeeds regardless of its ability to strategize, plan and execute?
  - samrus 1y ago
    it is. the benchmark was somewhat cheated, from the perspective of finding out how the model adjusts and plans within a dynamic reactive environment
  - 11101010001100 1y ago
    They asked gemini to come up with another word for cheating and it came up with 'harness'.
  - silvr 1y ago
    While true to a degree, I think this is largely wrong. Wouldn't it still count as a "harness" if we provided these LLMs with full robotic control of two humanoid arms, so that it could hold a Gameboy and play the game that way? I don't think the lack of that level of human-ness takes away from the demonstration of long-context reasoning that the GPP stream showed. Claude got stuck reasoning its way through one of the more complex puzzle areas. Gemini took a while on it also, but made it through. I don't that difference can be fully attributed up to the harnesses. Obviously, the best thing to do would be to run a SxS in the same harness of the two models. Maybe that will happen?
    - throwaway314155 1y ago
      I can appreciate that the model is likely still highly capable with a good harness. Still, I think this is more in line with ideas from say, speed running (or hell even reinforcement learning) where you want to prove something profound is possible and to do so before others do, you need to accumulate a series of "tricks" (refining exploits/hacking rewards) in order to achieve the goal. but if you use too many tricks you're no longer proving something as profound as originally claimed. In speed running this tends to splinter into multiple categories. Basically, the gane being conpleted by gemini was in an inferior category (however minuscule) of experiment. I get it though. People demanded these types of changes in the CPP twitch chat, because the pain of watching the model fail in slow motion is simply too much.
klohto 1y ago
2 weeks ago

minimaxir 1y ago

That Google IO slide was somewhat misleading as the maintainer of Gemini Plays Pokemon had a much better agentic harness that was constantly iterated upon throughout the runtime (e.g. the maintainer had to give specific instructions on how to use Strength to get past Victory Road), unlike Claude Plays Pokemon. The Elite Four/Champion was a non-issue in comparison especially when you have a lv. 81 Blastoise.

fourier456 1y ago
Okay, wait though like I want to know the full transcript because that actually is a better / softer benchmark if you measure in terms of the necessary human input.

hansmayer 1y ago

Right, but on the other hand... how is it even useful? Let's say it can beat the game, so what? So it can (kind of) summarise or write my emails - which is something I neither want nor need, they produce mountains of sloppy code, which I would have to end up fixing, and finally they can play a game? Where is the killer app? The gaming approach was exactly the premise of the original AI efforts in the 1960s, that teaching computers to play chess and other 'brainy' games will somehow lead to development of real AI. It ended as we know in the AI nuclear winter.

minimaxir 1y ago
It's a fun benchmark, like simonw's pelican riding a bike. Sometimes fun is the best metric.
samrus 1y ago
from a foundational research perspective, the pokemon benchmark is one of the most important ones. these models are trained on a static task, text generation, which is to say the state they are operating in does not change as they operate. but now that they are out we are implicitly demanding they do dynamic tasks like coding, navigation, operating in a market, or playing games. this are tasks where your state changes as you operate an example would be that as these models predict the next word, the ground truth of any further words doesnt change. if it misinterprets the word bank in the sentence "i went to the bank" as a river bank rather than a financial bank, the later ground truth wont change, if it was talking about the visit to the financial bank before, it will still be talking about that regardless of the model's misinterpretation. But if a model takes a wrong turn on the road, or makes a weird buy in the stock market, the environment will react and change and suddenly, what it should have done as the n+1th move before isnt the right move anymore, it needs to figure out a route of the freeway first, or deal with the FOMO bullrush it caused by mistakenly buying alot of stock we need to push against these limits to set the stage for the next evolution of AI, RL based models that are trained in dynamic reactive environments in the first place
- hansmayer 1y ago
  Honestly I have no idea what is this supposed to mean, and the high verbosity of whatever it is trying to prove is not helping it. To repeat: We already tried making computers play games. Ever heard of Deep Blue, and ever heard of it again since the early 2000s?

archon1410 1y ago

Claude Plays Pokemon was the original concept and inspiration behind "Gemini Plays Pokemon". Gemini arguably only did better because it had access to a much better agent harness and was being actively developed during the run. See: https://www.lesswrong.com/posts/7mqp8uRnnPdbBzJZE/is-gemini-now-better-than-claude-at-pokemon https://www.lesswrong.com/posts/7mqp8uRnnPdbBzJZE/is-gemini-...

montebicyclelo 1y ago
Not sure "original concept" is quite right, given it had been tried earlier, e.g. here's a 2023 attempt to get gpt-4-vision to play pokemon, (it didn't really work, but it's clearly "the concept") https://x.com/sidradcliffe/status/1722355983643525427 https://x.com/sidradcliffe/status/1722355983643525427
- archon1410 1y ago
  I see, I wasn't aware of that. The earliest attempt I knew of was from May 2024,[1] while this gpt-4-vision attempt is from November 2023. I guess Claude Plays Pokemon was the first attempt that had any real success (won a badge), and got a lot of attention over its entertaining "chain-of-thought". [1] https://community.aws/content/2gbBSofaMK7IDUev2wcUbqQXTK6/can-claude-play-pokemon-kind-of https://community.aws/content/2gbBSofaMK7IDUev2wcUbqQXTK6/ca...
silvr 1y ago
I disagree - this is all an homage to Twitch Plays Pokemon, which was a noteworthy moment in internet culture/history. https://en.wikipedia.org/wiki/Twitch_Plays_Pok%C3%A9mon https://en.wikipedia.org/wiki/Twitch_Plays_Pok%C3%A9mon