2 ms·

> Could I hook up a SOTA model the 2D computer puzzle game Gruntz (1999) so that it can read it from screenshots and act on it through keyboard and mouse inputs

by rfgplk 1mo ago

> Could I hook up a SOTA model the 2D computer puzzle game Gruntz (1999) so that it can read it from screenshots and act on it through keyboard and mouse inputs in a way where it would learn how to play and progress through the game? I don't think so. I doubt we'd see any sign of progress in building an internal model of how the game works and the win states in its "thinking" tokens. You.. literally can? I have no idea what 90% of the people here are saying, it's like they've never even used one of these models before.

Alwayshasbeeb 1mo ago
Can you? Let's say you prompt it with "this is a puzzle computer game, your objective is to progress through its levels" plus the controls from the instruction manual and tie it to a vision + KB and mouse harness. Will it effectively create an internal model describing world objects and how they interact with each other, persist that so it doesn't get lost when it's context window gets filled up, then after it has sufficiently complete knowledge of the fundamentals after the tutorial levels successfully apply that model by making plans to solve the puzzles and execute them by clicking the right coordinates tied to the visual feedback? I highly doubt it. To me it often just looks like people are defining narrow search spaces (e.g by having all of the task complexity pre-digested by the harness design), pointing a brute force engine at them, spending 20 thousand dollars in compute and then saying "hey look, it can do anything!".
- anthonyrstevens 1mo ago
  Frontier models can do this.
- robotresearcher 1mo ago
  Well, Go is a pretty complex game, and AlphaGo RL’d its way to excellence just by playing the game like you describe.
  - bananaflag 1mo ago
    By training. When we access the API, we don't get to train the model, we just do inference on the already trained model.
    - robotresearcher 1mo ago
      Oh, I see. That’s a very different requirement. It’s not a technical limitation but a product decision to not allow training. An advantage of properly open source models is that you can train and tune them. It’s an interesting challenge though. I might start to tackle it by having the model write its own tool program(s) to play the game. It’s possible that the model could choose that strategy itself from a high level prompt alone.

by rfgplk 1mo ago

Alwayshasbeeb 1mo ago

Can you? Let's say you prompt it with "this is a puzzle computer game, your objective is to progress through its levels" plus the controls from the instruction manual and tie it to a vision + KB and mouse harness. Will it effectively create an internal model describing world objects and how they interact with each other, persist that so it doesn't get lost when it's context window gets filled up, then after it has sufficiently complete knowledge of the fundamentals after the tutorial levels successfully apply that model by making plans to solve the puzzles and execute them by clicking the right coordinates tied to the visual feedback? I highly doubt it. To me it often just looks like people are defining narrow search spaces (e.g by having all of the task complexity pre-digested by the harness design), pointing a brute force engine at them, spending 20 thousand dollars in compute and then saying "hey look, it can do anything!".

anthonyrstevens 1mo ago
Frontier models can do this.
robotresearcher 1mo ago
Well, Go is a pretty complex game, and AlphaGo RL’d its way to excellence just by playing the game like you describe.
- bananaflag 1mo ago
  By training. When we access the API, we don't get to train the model, we just do inference on the already trained model.
  - robotresearcher 1mo ago
    Oh, I see. That’s a very different requirement. It’s not a technical limitation but a product decision to not allow training. An advantage of properly open source models is that you can train and tune them. It’s an interesting challenge though. I might start to tackle it by having the model write its own tool program(s) to play the game. It’s possible that the model could choose that strategy itself from a high level prompt alone.