7 ms·
Show HN: LLM plays Pokémon (open sourced)
I built a bot that plays Pokémon FireRed. It can explore, battle, and respond to game events. Farthest I made it was Viridian Forest.
I paused development a couple months ago, but given the launch of ClaudePlaysPokemon, decided to open source!
- rererereferred 2y agoWhat's the appeal of Pokemon for these kind of things? I never see AI or Twitch chat playing other turn based games like Final Fantasy or Fire Emblem.
- hombre_fatal 2y agoCultural mass.
- yifanl 2y agoPokemon is extremely hard to completely brick a run short of actively releasing your entire box which is very appealing for an MVP run, and is also literally the biggest media franchise in the world, which is a very appealing for people seeking hype.
- mminer237 2y agoThey're all inspired by TwitchPlaysPokemon, who chose Pokémon cuz he personally liked it and because "Even when played very poorly it is difficult not to make progress in Pokémon". It doesn't have game overs or permadeath. Even when you lose battles, you typically get stronger.
- minimaxir 2y agoSee also: the AI Plays Pokemon project that went megaviral a year or so ago, using CNNs and RL instead of LLMs: https://github.com/PWhiddy/PokemonRedExperiments https://github.com/PWhiddy/PokemonRedExperiments > I believe that Claude Plays Pokemon isn't doing any of the memory parsing I spent a ton of time, they are just streaming the memory directly to Claude 3.7 and it is figuring it out It is implied they are using structured Pokemon data from the LLM and saving it as a knowledge base. That is the only way they can get live Pokemon party data to display in the UI: https://www.twitch.tv/claudeplayspokemon https://www.twitch.tv/claudeplayspokemon The AI Plays Pokemon project above does note some of the memory addresses where that data is contained, since it used that data to calculate the reward for the PPO.
- adenta 2y agoOn this page (https://excalidraw.com/#json=WrM9ViixPu2je5cVJZGCe,no_UoONhF6UxyMpTqltYkg https://excalidraw.com/#json=WrM9ViixPu2je5cVJZGCe,no_UoONhF...) linked from their twitch, it says: “ This info is all parsed directly from the RAM of the game, Claude Code is very good at this task”. I’m reading that as “we are pumping the RAM directly into the LLM”, but I could be mistaken.
- None4U 2y agoI suspect that means they wrote the memory parser using Claude (the Twitch description also mentions the LLM getting specific info)
- minimaxir 2y agoI agree that's ambigiously worded. For example, I'm not sure if Claude could identify "MT MOON B1F" from the RAM data alone since internally world map areas are only known by IDs, while AI Plays Pokemon did annotate the corresponding area with a human-readable name. https://github.com/PWhiddy/PokemonRedExperiments/blob/master/v2/map_data.json https://github.com/PWhiddy/PokemonRedExperiments/blob/master... Though this RAM data could be in Claude's training data.
- swyx 2y agohttps://www.latent.space/p/how-claude-plays-pokemon-was-made https://www.latent.space/p/how-claude-plays-pokemon-was-made more info here too
- kanzure 2y agoYou can also directly pull in the emulation state and map back to game source code, and then make a script for tool use (not shown here): https://github.com/pret/pokemon-reverse-engineering-tools/blob/652269b739710f873d3def461aa2aa36db9e68da/pokemontools/vba/vba.py#L595 https://github.com/pret/pokemon-reverse-engineering-tools/bl... Well I see on your page that you already saw the pret advice about memory extraction, hopefully the link is useful anyway.
- adenta 2y agoYeah, took a similar approach at https://github.com/adenta/fire_red_agent/blob/main/app/services/game/map_reader.rb https://github.com/adenta/fire_red_agent/blob/main/app/servi...
- ArlenBales 2y ago> To me, this is the future of TV. The future of television is watching bots play video games? What a sad future.
- farts_mckensy 2y agoWatchin AI robots fight each other gladiator style world be pretty cool.
- deleted 2y ago[deleted]
- adenta 2y agoYeah I think _a_ future of television might've been more apt. They should've made a season 5 to Snowpiercer.
- deadbabe 2y agoI want to note that if you really wanted an AI to play Pokémon you can do it with a far simpler and cheaper AI than an LLM and it would play the game far better, making this mostly an exercise in overcomplicating something trivial. But sometimes when you have a hammer everything will look like a nail.
- adenta 2y agoI disagree. Getting a computer to play a game like a human has an incredibly broad range of applications. Imagine a system like this that is on autopilot, but can get suggestions from a twitch chat, nudging its behavior in a specific direction. Two such systems could be run by two teams, and they could do a weekly battle. This isn’t an exercise in AI, it’s an exercise in TV production IMO.
- minimaxir 2y agoThe AI Plays Pokemon project only made it to Mt. Moon (where coincidentially ClaudePlaysPokemon is stuck now) with many months of iteration and many many hours of compute. The reason Claude 3.7's performance is interesting is that the LLM approach defeated Lt. Surge, far past Mt. Moon. (I wonder how Claude solved the infamous puzzle in Surge's gym) https://www.anthropic.com/research/visible-extended-thinking https://www.anthropic.com/research/visible-extended-thinking
- deadbabe 2y agoNot talking about Reinforcement learning type AI, I’m talking about classically programmed AI with standard pathfinders, GOAP, behavior trees, etc…
- dang 2y agoRelated ongoing thread: Claude Plays Pokémon - https://news.ycombinator.com/item?id=43173825 https://news.ycombinator.com/item?id=43173825
- tgtweak 2y agoYou can use claude computer functions to actually play it on an emulator with no programming at all - but that kind of feels like cheating :D
- adenta 2y agoI tried! It didn’t work super well
- mclau156 2y agoHonestly Claude 3.7 can make a pokemon game in pygame fairly easily, at that point it would have a lot more control over it
- genewitch 2y agoThis is what codebullet does, YouTube channel. Recreate games so that an agent can play them and try to win, high score, whatever.
- evanextreme 2y agoWas working on a similar thing last year! Might as well open source at this point too.
- adenta 2y agoEmail me when it launches! (In profile)
- podoman 2y agoHave you considered calling this bot "intern bot"? - Jay
- adenta 2y agohttps://static.wikia.nocookie.net/loveinterest/images/5/5c/3458_6_full.jpg/revision/latest?cb=20130526104055 https://static.wikia.nocookie.net/loveinterest/images/5/5c/3...
- montebicyclelo 2y agoSuper cool to see this idea working. I had a go at getting an LLM to play Pokémon in 2023, with openai vision. With only 100 expensive api calls a day, I shelved the project after putting together a quick POC and finding that the model struggled to see things or work out where the player was. I guess models are now better, but also looks like people are providing the model with information in addition to the game screen. https://x.com/sidradcliffe/status/1722355983643525427?t=dYMk2oZ90pEokFzQrAKoaw&s=19 https://x.com/sidradcliffe/status/1722355983643525427?t=dYMk...
- LZ_Khan 2y agoI just find it insane that we're bootstrapping reinforcement learning and world planning on top of basic next token prediction. I'm amazed that it works, but also amazed that this is the approach being prioritized.
- AustinDev 2y agoPure RL NN 'solved' simple games like Pokémon years ago. I think added challenge of seeing how well LLMs can generalize is a noble pursuit. I think games are a fun problem as well. Look how poorly Claude 3.7 is doing on Pokemon on Twitch right now.
- minimaxir 2y ago> Pure RL NN 'solved' simple games like Pokémon years ago. Please link to said project. From my search of Google filtered to 2010-2020, it returns nothing outside of proofs-of-concept (e.g. https://github.com/poke-AI/poke.AI https://github.com/poke-AI/poke.AI) that do not perform any better, or instead trying to solve Pokemon battles which are an order of magnitude easier.
- rockwotj 2y agoMaybe they are conflating the Starcraft success that Deepmind had with AlphaStar?
- linwangg 2y agoReally cool experiment! The idea of AI 'playing' games as a form of entertainment is fascinating—kind of like Twitch streams but fully autonomous. Curious, what were the biggest hurdles with input control? Lag, accuracy, or something else?
- adenta 2y agoI couldn't get https://docs.libretro.com/library/remote_retropad/ https://docs.libretro.com/library/remote_retropad/ to work! its designed for game pads to control the emulator. I was hoping I could repurpose it to just send arbitrary key commands over a network interface, but nothing I tried worked. When I asked in the retroarch discord, they couldn't figure it out either lol.
- phatskat 2y agoI doubt this answers any of your questions but check out saltybet on twitch - people make fighters and the computer pits them against each other a la street fighter