4 ms·

Nice writeup! I need to start blogging about my antics. I rigged up several cutting edge small local models to an emulator all in-browser and unsuccessfully tri

Nice writeup! I need to start blogging about my antics. I rigged up several cutting edge small local models to an emulator all in-browser and unsuccessfully tried to get them to play different Pokémon games. They just weren't as sharp as the frontier models. This was a good while back but I'm sure a lot of people might find the process and code interesting even if it didn't succeed. Might resurrect that project.

giancarlostoro 10mo ago
I have to think they need to know enough of the guides for the game for it to work out, how do they know whats on screen?
- soulofmischief 10mo ago
  In my project I rigged up an in-browser emulator and directly fed captured images of the screen to local multimodal models. So it just looks right at what's going on, writes a description for refinement, and uses all of that to create and manage goals, write to a scratchpad and submit input. It's minimal scaffolding because I wanted to see what these raw models are capable of. Kind of a benchmark.
  - giancarlostoro 10mo ago
    I have a feeling if you gave them access to GameFAQ guides they might be able to play better, but it depends on how you can feed them the data.
    - soulofmischief 10mo ago
      It turns out that cutting edge super small (3b param etc) models that fit in the browser are not great at playing Pokémon on an even basic level, even navigation is difficult when only providing raw visual information, and object recognition of the low-resolution sprites is not great. So I lost interest before even getting to the point of providing specific strategy. But, it runs in browser and works with any supplied ROM, none of it is Pokémon-specific so I should set aside time to serve it and make the code available
      - giancarlostoro 10mo ago
        You should consider publishing your setup as either a blog post or a GitHub repo. I think it could make for good benchmarking of smaller models. Ideally we can one day all run small models that can do amazing things.

giancarlostoro 10mo ago

I have to think they need to know enough of the guides for the game for it to work out, how do they know whats on screen?

soulofmischief 10mo ago
In my project I rigged up an in-browser emulator and directly fed captured images of the screen to local multimodal models. So it just looks right at what's going on, writes a description for refinement, and uses all of that to create and manage goals, write to a scratchpad and submit input. It's minimal scaffolding because I wanted to see what these raw models are capable of. Kind of a benchmark.
- giancarlostoro 10mo ago
  I have a feeling if you gave them access to GameFAQ guides they might be able to play better, but it depends on how you can feed them the data.
  - soulofmischief 10mo ago
    It turns out that cutting edge super small (3b param etc) models that fit in the browser are not great at playing Pokémon on an even basic level, even navigation is difficult when only providing raw visual information, and object recognition of the low-resolution sprites is not great. So I lost interest before even getting to the point of providing specific strategy. But, it runs in browser and works with any supplied ROM, none of it is Pokémon-specific so I should set aside time to serve it and make the code available
    - giancarlostoro 10mo ago
      You should consider publishing your setup as either a blog post or a GitHub repo. I think it could make for good benchmarking of smaller models. Ideally we can one day all run small models that can do amazing things.