4 ms·

Could I hook up a SOTA model the 2D computer puzzle game Gruntz (1999) so that it can read it from screenshots and act on it through keyboard and mouse inputs i

Could I hook up a SOTA model the 2D computer puzzle game Gruntz (1999) so that it can read it from screenshots and act on it through keyboard and mouse inputs in a way where it would learn how to play and progress through the game? I don't think so. I doubt we'd see any sign of progress in building an internal model of how the game works and the win states in its "thinking" tokens. Anyone that can read English could do that though.

scotty79 1mo ago
Isn't that what ARC-AGI-3 was trying to do and Astra basically just aced it if it was not given amnesia every turn?
vb-8448 1mo ago
But someone could probably build a harness what will be able to do play the game.
10xDev 1mo ago
This statement is behind times, go and watch astra play Pokemon: https://www.twitch.tv/gpt_plays_pokemon https://www.twitch.tv/gpt_plays_pokemon
- kcatskcolbdi 1mo ago
  It can't even sprint because it's incapable of pressing two buttons at the same time. Maybe we get the two button tech before we start celebrating AGI.
rfgplk 1mo ago
> Could I hook up a SOTA model the 2D computer puzzle game Gruntz (1999) so that it can read it from screenshots and act on it through keyboard and mouse inputs in a way where it would learn how to play and progress through the game? I don't think so. I doubt we'd see any sign of progress in building an internal model of how the game works and the win states in its "thinking" tokens. You.. literally can? I have no idea what 90% of the people here are saying, it's like they've never even used one of these models before.
- Alwayshasbeeb 1mo ago
  Can you? Let's say you prompt it with "this is a puzzle computer game, your objective is to progress through its levels" plus the controls from the instruction manual and tie it to a vision + KB and mouse harness. Will it effectively create an internal model describing world objects and how they interact with each other, persist that so it doesn't get lost when it's context window gets filled up, then after it has sufficiently complete knowledge of the fundamentals after the tutorial levels successfully apply that model by making plans to solve the puzzles and execute them by clicking the right coordinates tied to the visual feedback? I highly doubt it. To me it often just looks like people are defining narrow search spaces (e.g by having all of the task complexity pre-digested by the harness design), pointing a brute force engine at them, spending 20 thousand dollars in compute and then saying "hey look, it can do anything!".
  - anthonyrstevens 1mo ago
    Frontier models can do this.
  - robotresearcher 1mo ago
    Well, Go is a pretty complex game, and AlphaGo RL’d its way to excellence just by playing the game like you describe.
    - bananaflag 1mo ago
      By training. When we access the API, we don't get to train the model, we just do inference on the already trained model.
      - robotresearcher 1mo ago
        Oh, I see. That’s a very different requirement. It’s not a technical limitation but a product decision to not allow training. An advantage of properly open source models is that you can train and tune them. It’s an interesting challenge though. I might start to tackle it by having the model write its own tool program(s) to play the game. It’s possible that the model could choose that strategy itself from a high level prompt alone.
beering 1mo ago
5.6 Sol can already do this with two caveats: 1. It’s too slow for real-time games. To play mario, you’d need to step frame by frame like a TAS. I don’t know if Gruntz has real-time elements or not. 2. It will be expensive. You won’t get very far with a Plus subscription. The models likely already have some knowledge on game objectives unless the game is really obscure, so it should do a decent job. It can figure out details of the mechanics along the way.

scotty79 1mo ago

Isn't that what ARC-AGI-3 was trying to do and Astra basically just aced it if it was not given amnesia every turn?

vb-8448 1mo ago

But someone could probably build a harness what will be able to do play the game.

10xDev 1mo ago

This statement is behind times, go and watch astra play Pokemon: https://www.twitch.tv/gpt_plays_pokemon https://www.twitch.tv/gpt_plays_pokemon

kcatskcolbdi 1mo ago
It can't even sprint because it's incapable of pressing two buttons at the same time. Maybe we get the two button tech before we start celebrating AGI.

rfgplk 1mo ago

> Could I hook up a SOTA model the 2D computer puzzle game Gruntz (1999) so that it can read it from screenshots and act on it through keyboard and mouse inputs in a way where it would learn how to play and progress through the game? I don't think so. I doubt we'd see any sign of progress in building an internal model of how the game works and the win states in its "thinking" tokens. You.. literally can? I have no idea what 90% of the people here are saying, it's like they've never even used one of these models before.

Alwayshasbeeb 1mo ago
Can you? Let's say you prompt it with "this is a puzzle computer game, your objective is to progress through its levels" plus the controls from the instruction manual and tie it to a vision + KB and mouse harness. Will it effectively create an internal model describing world objects and how they interact with each other, persist that so it doesn't get lost when it's context window gets filled up, then after it has sufficiently complete knowledge of the fundamentals after the tutorial levels successfully apply that model by making plans to solve the puzzles and execute them by clicking the right coordinates tied to the visual feedback? I highly doubt it. To me it often just looks like people are defining narrow search spaces (e.g by having all of the task complexity pre-digested by the harness design), pointing a brute force engine at them, spending 20 thousand dollars in compute and then saying "hey look, it can do anything!".
- anthonyrstevens 1mo ago
  Frontier models can do this.
- robotresearcher 1mo ago
  Well, Go is a pretty complex game, and AlphaGo RL’d its way to excellence just by playing the game like you describe.
  - bananaflag 1mo ago
    By training. When we access the API, we don't get to train the model, we just do inference on the already trained model.
    - robotresearcher 1mo ago
      Oh, I see. That’s a very different requirement. It’s not a technical limitation but a product decision to not allow training. An advantage of properly open source models is that you can train and tune them. It’s an interesting challenge though. I might start to tackle it by having the model write its own tool program(s) to play the game. It’s possible that the model could choose that strategy itself from a high level prompt alone.

beering 1mo ago

5.6 Sol can already do this with two caveats: 1. It’s too slow for real-time games. To play mario, you’d need to step frame by frame like a TAS. I don’t know if Gruntz has real-time elements or not. 2. It will be expensive. You won’t get very far with a Plus subscription. The models likely already have some knowledge on game objectives unless the game is really obscure, so it should do a decent job. It can figure out details of the mechanics along the way.