6 ms·

Baba Is Eval

254 points by fi-le 1y ago

paxcoder 1y ago
[dead]
kinduff 1y ago
Do you think the performance can be improved if the representation of the level is different? I've seen AI struggle with ASCII, but when presented as other data structures, it performs better. edit: e.g. JSON with structured coordinates, graph based JSON, or a semantic representation with the coordinates
- hajile 1y ago
  If it struggles with the representation, that makes it an even better test of the AI's thinking potential.
  - eru 1y ago
    I'm not sure. Adding superficial difficulties to an IQ test for humans doesn't (necessarily) improve it as an IQ test.
- RainyDayTmrw 1y ago
  In the limit case, to an actual general intelligence, representation is superfluous, because it can figure out how to convert freely. To the extent that the current generation of AI isn't general, yeah, papering over some of its weaknesses may allow you to expose other parts of it, both strengths and other weaknesses.
  - kadoban 1y ago
    A human can easily struggle at solving a poorly communicated puzzle, especially if paper/pencil or something isn't available to convert to a better format. LLMs can look back at what they wrote, but it seems kind of like a poor format for working out a better representation to me.
    - kinduff 1y ago
      I found some papers [n] about this. And I think the answer is yes, the format matters asnd hence the representation. I wonder if the author would be willing to try with another representation. [1]: Does Prompt Formatting Have Any Impact on LLM Performance? https://arxiv.org/html/2411.10541v1 https://arxiv.org/html/2411.10541v1 [2]: Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding - A Survey https://arxiv.org/html/2402.17944v2 https://arxiv.org/html/2402.17944v2
- QuadmasterXLII 1y ago
  These models can “code,” but they can’t code yet. We’ll know that they can actually code once their performance on these tasks becomes invariant to input representation, because they can just whip up a script to convert representations.
k2xl 1y ago
Baba is You is a great game part of a collection of 2D grid puzzle games. (Shameless plug: I am one of the developers of Thinky.gg (https://thinky.gg https://thinky.gg), which is a thinky puzzle game site for a 'shortest path style' [Pathology] and a Sokoban variant [Sokoath] ) These games are typically NP Hard so the typical techniques that solvers have employed for Sokoban (or Pathology) have been brute forced with varying heuristics (like BFS, dead-lock detection, and Zobrist hashing). However, once levels get beyond a certain size with enough movable blocks you end up exhausting memory pretty quickly. These types of games are still "AI Proof" so far in that LLMs are absolutely awful at solving these while humans are very good (so seems reasonable to consider for for ARC-AGI benchmarks). Whenever a new reasoning model gets released I typically try it on some basic Pathology levels (like 'One at a Time' https://pathology.thinky.gg/level/ybbun/one-at-a-time https://pathology.thinky.gg/level/ybbun/one-at-a-time) and they fail miserably. Simple level code for the above level (1 is a wall, 2 is a movable block, 4 is starting block, 3 is the exit): 000 020 023 041 Similar to OP, I've found Claude couldn’t manage rule dynamics, blocked paths, or game objectives well and spits out random results.
- kinduff 1y ago
  In Factorio's paper [1] page 3, the agent receives a semantic representation with coordinates. Have you tried this data format? [1]: https://arxiv.org/pdf/2503.09617 https://arxiv.org/pdf/2503.09617
- eru 1y ago
  NP hard isn't much of a problem, because the levels are fairly small, and instances are not chosen to be worst case hard but to be entertaining for humans to solve. SMT/SAT solvers or integer linear programming can get you pretty far. Many classic puzzle games like Minesweeper are NP hard, and you can solve any instance that a human would be able to solve in their lifetime fairly quickly on a computer.
ekianjo 1y ago
this is definitely a case for fine tuning a LLM on this game's data. There is currently no LLM out there that is able to play very well many games of different kinds.
captn3m0 1y ago
I once made a “RC plays Baba Is You” that controlled the game over a single shared browser that was streaming video and controls back to the game. Was quite fun! But I am fairly sure all of Baba Is You solutions are present in the training data for modern LLMs so it won’t make for a good eval.
- chmod775 1y ago
  > But I am fairly sure all of Baba Is You solutions are present in the training data for modern LLMs so it won’t make for a good eval. Claude 4 cannot solve any Baba Is You level (except level 0 that is solved by 8 right inputs), so for now it's at least a nice low bar to shoot for...
RainyDayTmrw 1y ago
This is interesting. If you approach this game as individual moves, the search tree is really deep. However, most levels can be expressed as a few intermediate goals. In some ways, this reminds me of the history of AI Go (board game). But the resolution there was MCTS, which wasn't at all what we wanted (insofar as MCTS is not generalizable to most things).
- rtpg 1y ago
  > However, most levels can be expressed as a few intermediate goals I think generally the whole thing with puzzle games is that you have to determine the “right” intermediate goals. In fact, the naive intermediate goals are often entirely wrong! A canonical sokoban-like inversion might be where you have to push two blocks into goal areas. You might think “ok, push one block into its goal area and then push another into it.” But many of these games will have mechanisms meaning you would first want to push one block into its goal, then undo that for some reason (it might activate some extra functionality) push the other block, and then finally go back and do the thing. There’s always weird tricks that mean that you’re going to walk backwards before walking forwards. I don’t think it’s impossible for these things to stumble into it, though. Just might spin a lot of cycles to get there (humans do too I guess)
  - matsemann 1y ago
    Yeah, often working backwards and forwards at the same time is how to solve some advanced puzzle games. Then you keep it from exploding in options. When thinking backwards from the goal, you figure out constraints or "invariants" the forward path must uphold, thus can discard lots of dead ends earlier in your forward path. To me, those discoveries are the fun part of most puzzle games. When you unlock the "trick" for each level and the dopamine flies, heh.
    - TeMPOraL 1y ago
      I usually get a good mileage out of jumping straight in the middle :). Like, "hmm let's look at this block; oh cool, there's enough space around it that I could push it away from goal, for whatever reason". Turns out, if it's possible there usually is a good reason. So whenever I get stuck, I skim every object in the puzzle and consider in isolation, what can I do with it, and this usually gives me anchor points to drive my forward or backward thinking through.
pclmulqdq 1y ago
I have noticed a trend of the word "Desiderata" appearing in a lot more writing. Is this an LLM word or is it just in fashion? Most people would use the words "Deisres" or "Goals," so I assume this might be the new "delve."
- Tomte 1y ago
  It‘s academic jargon. Desiderata are often at the end of a paper, in the section „someone should investigate X, but I‘m moving on to the next funded project“.
  - ginko 1y ago
    So „Future Work“?
    - dgfl 1y ago
      Literally it means “things that we wish for”, from the latin verb “desiderare” (to wish).
- fi-le 1y ago
  At least in this instance, it came from my fleshy human brain. Although I perhaps used it to come off as smarter than I really am - just like an LLM might.
b0a04gl 1y ago
[dead]
wohoef 1y ago
In my experience LLMs have a hard time working with text grids like this. It seems to find columns harder to “detect” then rows. Probably because it’s input shows it as a giant row if that makes sense. It has the same problem with playing chess. But I’m not sure if there is a datatype it could work with for this kinda game. Currently it seems more like LLMs can’t really work on spacial problems. But this should actually be something that can be fixed (pretty sure I saw an article about it on HN recently)
- froobius 1y ago
  Transformers can easily be trained / designed to handle grids, it's just that off the shelf standard LLMs haven't been particularly, (although they would have seen some)
  - nine_k 1y ago
    Are there some well-known examples of success in it?
    - thethimble 1y ago
      Vision transformers effectively encode a grid of pixel patches. It’s ultimately a matter of ensuring the position encoding incorporates both X and Y and position. For LLMs we only have one axis of position and - more importantly - the vast majority of training data only is oriented in this way.
- stavros 1y ago
  If this were a limitation in the architecture, they wouldn't be able to work with images, no?
  - hnlmorg 1y ago
    LLMs don’t work with images.
    - 1y ago
tibastral2 1y ago
It reminds me of https://en.m.wikipedia.org/wiki/The_Ricks_Must_Be_Crazy https://en.m.wikipedia.org/wiki/The_Ricks_Must_Be_Crazy. Hope we are not ourselves in some sort of simulation ;)
ThouTo2C 1y ago
There are numerous guides for all levels of Baba Is You available. I think it's likely that any modern LLM has them as part of its training dataset. That severely degrades this as a test for complex solution capabilities. Still, its interesting to see the challenges with dynamic rules (like "Key is Stop") that change where are you able to move etc.
- klohto 1y ago
  Read the article first maybe
- ethan_smith 1y ago
  The dynamic rule changes are precisely what make this a valuable benchmark despite available guides. Each rule modification creates a novel state-space that requires reasoning about the consequences of those changes, not just memorizing solution paths.
niemandhier 1y ago
I think it’s a great idea for a benchmark. One key difference to ARC in its current iteration is that there is a defined and learnable game physics. Arc requires generalization based on few examples for problems that are not well defined per se. Hence ARC currently requires the models that work on it to possess biases that are comparable to the ones that humans possess.
andy99 1y ago
I suspect real AGI evals aren't going to be "IQ test"-like which is how I'd categorize these benchmarks. LLMs will probably continue to scale on such benchmarks, as they have been, without needing real ingenuity or intelligence. Obviously I don't know the answer but I think it's the same root problem as why neural networks will never lead to intelligence. We're building and testing idiot savants.
popcar2 1y ago
I would be way more interested in it playing niche community levels, because I suspect a huge reason it's able to solve these levels is because it was trained on a million Baba is You walkthroughs. Same with people using Pokemon as a way to test LLMs, it really just depends on how well it knows the game.
- deleted 1y ago
  [deleted]
- fi-le 1y ago
  Two corrections, as written in the post: At least Claude not able to solve the standard levels at all, and community levels are definitely in scope.
WhitneyLand 1y ago
“Reasoning models like o3 might be better equipped to come up with a plan, so a natural step would be to try switching to those, away from Claude Desktop…” But…Claude Desktop does have a reasoning mode for both Sonnet and Opus.
zahlman 1y ago
> This is why the video of Claude solving level 1 at the top was actually (dramatic musical cue) staged, and only possible via a move-for-move tutorial that Claude nicely rationalized post hoc. One of the things this arc of history has taught me is that post-hoc rationalization is depressingly easy. Especially if it doesn't have to make sense, but even passing basic logical checks isn't too difficult. Ripping the rationalization apart often requires identifying novel, non-obvious logical checks. I thought I had learned that time and time again from human politics, but AI somehow made it even clearer than I thought possible. Perhaps simply because of knowing that a machine is doing it. Edit: after watching the video more carefully: > "This forms WALL IS WIN horizontally. But I need "FLAG IS WIN" instead. Let me check if walls now have the WIN property. If they do, I just need to touch a wall to win. Let me try moving to a wall: There's something extremely uncanny-valley about this. A human player absolutely would accidentally win like this, and have similar reasoning (not expressed so formally) about how the win was achieved after the fact. (Winning depends on the walls having WIN and also not having STOP; many players get stuck on later levels, even after having supposedly learned the lesson of this one, by trying to make something WIN and walk onto it while it is still STOP.) But the WIN block was not originally in line with the WALL IS text, so a human player would never accidentally form the rule, but would only do it with the expectation of being able to win that way. Especially since there was already an obvious, clear path to FLAG — a level like this has no Sokoban puzzle element to it; it's purely about learning that the walls only block the player because they are STOP. Nor would (from my experience watching streamers at least) a human spontaneously notice that the rule "WALL IS WIN" had been formed and treat that as a cue to reconsider the entire strategy. The natural human response to unintentionally forming a useful rule is to keep pushing in the same direction. On the other hand, an actually dedicated AI system (in the way that AlphaGo was dedicated to Go) could, I'm sure, figure out a game like Baba Is You pretty easily. It would lack the human instinct to treat the walls as if they were implicitly always STOP; so it would never struggle with overriding it.
- deadbabe 1y ago
  A simple feed-forward neural network with sufficient training can solve levels way better than Claude. Why is Claude being used at all.
dezgeg 1y ago
Cool, it's something I've been trying also. Here are my attempts: https://www.youtube.com/watch?v=JZ4-YUuCVNo&list=PL8C_UWcLmvGlv7dWY-oSowUfgFnDOeDg2 https://www.youtube.com/watch?v=JZ4-YUuCVNo&list=PL8C_UWcLmv...

254 points by fi-le 1y ago

[dead]

Do you think the performance can be improved if the representation of the level is different? I've seen AI struggle with ASCII, but when presented as other data structures, it performs better. edit: e.g. JSON with structured coordinates, graph based JSON, or a semantic representation with the coordinates

hajile 1y ago
If it struggles with the representation, that makes it an even better test of the AI's thinking potential.
- eru 1y ago
  I'm not sure. Adding superficial difficulties to an IQ test for humans doesn't (necessarily) improve it as an IQ test.
RainyDayTmrw 1y ago
In the limit case, to an actual general intelligence, representation is superfluous, because it can figure out how to convert freely. To the extent that the current generation of AI isn't general, yeah, papering over some of its weaknesses may allow you to expose other parts of it, both strengths and other weaknesses.
- kadoban 1y ago
  A human can easily struggle at solving a poorly communicated puzzle, especially if paper/pencil or something isn't available to convert to a better format. LLMs can look back at what they wrote, but it seems kind of like a poor format for working out a better representation to me.
  - kinduff 1y ago
    I found some papers [n] about this. And I think the answer is yes, the format matters asnd hence the representation. I wonder if the author would be willing to try with another representation. [1]: Does Prompt Formatting Have Any Impact on LLM Performance? https://arxiv.org/html/2411.10541v1 https://arxiv.org/html/2411.10541v1 [2]: Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding - A Survey https://arxiv.org/html/2402.17944v2 https://arxiv.org/html/2402.17944v2
QuadmasterXLII 1y ago
These models can “code,” but they can’t code yet. We’ll know that they can actually code once their performance on these tasks becomes invariant to input representation, because they can just whip up a script to convert representations.

k2xl 1y ago

Baba is You is a great game part of a collection of 2D grid puzzle games. (Shameless plug: I am one of the developers of Thinky.gg (https://thinky.gg https://thinky.gg), which is a thinky puzzle game site for a 'shortest path style' [Pathology] and a Sokoban variant [Sokoath] ) These games are typically NP Hard so the typical techniques that solvers have employed for Sokoban (or Pathology) have been brute forced with varying heuristics (like BFS, dead-lock detection, and Zobrist hashing). However, once levels get beyond a certain size with enough movable blocks you end up exhausting memory pretty quickly. These types of games are still "AI Proof" so far in that LLMs are absolutely awful at solving these while humans are very good (so seems reasonable to consider for for ARC-AGI benchmarks). Whenever a new reasoning model gets released I typically try it on some basic Pathology levels (like 'One at a Time' https://pathology.thinky.gg/level/ybbun/one-at-a-time https://pathology.thinky.gg/level/ybbun/one-at-a-time) and they fail miserably. Simple level code for the above level (1 is a wall, 2 is a movable block, 4 is starting block, 3 is the exit): 000 020 023 041 Similar to OP, I've found Claude couldn’t manage rule dynamics, blocked paths, or game objectives well and spits out random results.

kinduff 1y ago
In Factorio's paper [1] page 3, the agent receives a semantic representation with coordinates. Have you tried this data format? [1]: https://arxiv.org/pdf/2503.09617 https://arxiv.org/pdf/2503.09617
eru 1y ago
NP hard isn't much of a problem, because the levels are fairly small, and instances are not chosen to be worst case hard but to be entertaining for humans to solve. SMT/SAT solvers or integer linear programming can get you pretty far. Many classic puzzle games like Minesweeper are NP hard, and you can solve any instance that a human would be able to solve in their lifetime fairly quickly on a computer.

ekianjo 1y ago

this is definitely a case for fine tuning a LLM on this game's data. There is currently no LLM out there that is able to play very well many games of different kinds.

captn3m0 1y ago

I once made a “RC plays Baba Is You” that controlled the game over a single shared browser that was streaming video and controls back to the game. Was quite fun! But I am fairly sure all of Baba Is You solutions are present in the training data for modern LLMs so it won’t make for a good eval.

chmod775 1y ago
> But I am fairly sure all of Baba Is You solutions are present in the training data for modern LLMs so it won’t make for a good eval. Claude 4 cannot solve any Baba Is You level (except level 0 that is solved by 8 right inputs), so for now it's at least a nice low bar to shoot for...

RainyDayTmrw 1y ago

This is interesting. If you approach this game as individual moves, the search tree is really deep. However, most levels can be expressed as a few intermediate goals. In some ways, this reminds me of the history of AI Go (board game). But the resolution there was MCTS, which wasn't at all what we wanted (insofar as MCTS is not generalizable to most things).

rtpg 1y ago
> However, most levels can be expressed as a few intermediate goals I think generally the whole thing with puzzle games is that you have to determine the “right” intermediate goals. In fact, the naive intermediate goals are often entirely wrong! A canonical sokoban-like inversion might be where you have to push two blocks into goal areas. You might think “ok, push one block into its goal area and then push another into it.” But many of these games will have mechanisms meaning you would first want to push one block into its goal, then undo that for some reason (it might activate some extra functionality) push the other block, and then finally go back and do the thing. There’s always weird tricks that mean that you’re going to walk backwards before walking forwards. I don’t think it’s impossible for these things to stumble into it, though. Just might spin a lot of cycles to get there (humans do too I guess)
- matsemann 1y ago
  Yeah, often working backwards and forwards at the same time is how to solve some advanced puzzle games. Then you keep it from exploding in options. When thinking backwards from the goal, you figure out constraints or "invariants" the forward path must uphold, thus can discard lots of dead ends earlier in your forward path. To me, those discoveries are the fun part of most puzzle games. When you unlock the "trick" for each level and the dopamine flies, heh.
  - TeMPOraL 1y ago
    I usually get a good mileage out of jumping straight in the middle :). Like, "hmm let's look at this block; oh cool, there's enough space around it that I could push it away from goal, for whatever reason". Turns out, if it's possible there usually is a good reason. So whenever I get stuck, I skim every object in the puzzle and consider in isolation, what can I do with it, and this usually gives me anchor points to drive my forward or backward thinking through.

pclmulqdq 1y ago

I have noticed a trend of the word "Desiderata" appearing in a lot more writing. Is this an LLM word or is it just in fashion? Most people would use the words "Deisres" or "Goals," so I assume this might be the new "delve."

Tomte 1y ago
It‘s academic jargon. Desiderata are often at the end of a paper, in the section „someone should investigate X, but I‘m moving on to the next funded project“.
- ginko 1y ago
  So „Future Work“?
  - dgfl 1y ago
    Literally it means “things that we wish for”, from the latin verb “desiderare” (to wish).
fi-le 1y ago
At least in this instance, it came from my fleshy human brain. Although I perhaps used it to come off as smarter than I really am - just like an LLM might.

b0a04gl 1y ago

[dead]

wohoef 1y ago

In my experience LLMs have a hard time working with text grids like this. It seems to find columns harder to “detect” then rows. Probably because it’s input shows it as a giant row if that makes sense. It has the same problem with playing chess. But I’m not sure if there is a datatype it could work with for this kinda game. Currently it seems more like LLMs can’t really work on spacial problems. But this should actually be something that can be fixed (pretty sure I saw an article about it on HN recently)

froobius 1y ago
Transformers can easily be trained / designed to handle grids, it's just that off the shelf standard LLMs haven't been particularly, (although they would have seen some)
- nine_k 1y ago
  Are there some well-known examples of success in it?
  - thethimble 1y ago
    Vision transformers effectively encode a grid of pixel patches. It’s ultimately a matter of ensuring the position encoding incorporates both X and Y and position. For LLMs we only have one axis of position and - more importantly - the vast majority of training data only is oriented in this way.
stavros 1y ago
If this were a limitation in the architecture, they wouldn't be able to work with images, no?
- hnlmorg 1y ago
  LLMs don’t work with images.
  - 1y ago

tibastral2 1y ago

It reminds me of https://en.m.wikipedia.org/wiki/The_Ricks_Must_Be_Crazy https://en.m.wikipedia.org/wiki/The_Ricks_Must_Be_Crazy. Hope we are not ourselves in some sort of simulation ;)

ThouTo2C 1y ago

There are numerous guides for all levels of Baba Is You available. I think it's likely that any modern LLM has them as part of its training dataset. That severely degrades this as a test for complex solution capabilities. Still, its interesting to see the challenges with dynamic rules (like "Key is Stop") that change where are you able to move etc.

klohto 1y ago
Read the article first maybe
ethan_smith 1y ago
The dynamic rule changes are precisely what make this a valuable benchmark despite available guides. Each rule modification creates a novel state-space that requires reasoning about the consequences of those changes, not just memorizing solution paths.

niemandhier 1y ago

I think it’s a great idea for a benchmark. One key difference to ARC in its current iteration is that there is a defined and learnable game physics. Arc requires generalization based on few examples for problems that are not well defined per se. Hence ARC currently requires the models that work on it to possess biases that are comparable to the ones that humans possess.

andy99 1y ago

I suspect real AGI evals aren't going to be "IQ test"-like which is how I'd categorize these benchmarks. LLMs will probably continue to scale on such benchmarks, as they have been, without needing real ingenuity or intelligence. Obviously I don't know the answer but I think it's the same root problem as why neural networks will never lead to intelligence. We're building and testing idiot savants.

popcar2 1y ago

I would be way more interested in it playing niche community levels, because I suspect a huge reason it's able to solve these levels is because it was trained on a million Baba is You walkthroughs. Same with people using Pokemon as a way to test LLMs, it really just depends on how well it knows the game.

deleted 1y ago
[deleted]
fi-le 1y ago
Two corrections, as written in the post: At least Claude not able to solve the standard levels at all, and community levels are definitely in scope.

WhitneyLand 1y ago

“Reasoning models like o3 might be better equipped to come up with a plan, so a natural step would be to try switching to those, away from Claude Desktop…” But…Claude Desktop does have a reasoning mode for both Sonnet and Opus.

zahlman 1y ago

> This is why the video of Claude solving level 1 at the top was actually (dramatic musical cue) staged, and only possible via a move-for-move tutorial that Claude nicely rationalized post hoc. One of the things this arc of history has taught me is that post-hoc rationalization is depressingly easy. Especially if it doesn't have to make sense, but even passing basic logical checks isn't too difficult. Ripping the rationalization apart often requires identifying novel, non-obvious logical checks. I thought I had learned that time and time again from human politics, but AI somehow made it even clearer than I thought possible. Perhaps simply because of knowing that a machine is doing it. Edit: after watching the video more carefully: > "This forms WALL IS WIN horizontally. But I need "FLAG IS WIN" instead. Let me check if walls now have the WIN property. If they do, I just need to touch a wall to win. Let me try moving to a wall: There's something extremely uncanny-valley about this. A human player absolutely would accidentally win like this, and have similar reasoning (not expressed so formally) about how the win was achieved after the fact. (Winning depends on the walls having WIN and also not having STOP; many players get stuck on later levels, even after having supposedly learned the lesson of this one, by trying to make something WIN and walk onto it while it is still STOP.) But the WIN block was not originally in line with the WALL IS text, so a human player would never accidentally form the rule, but would only do it with the expectation of being able to win that way. Especially since there was already an obvious, clear path to FLAG — a level like this has no Sokoban puzzle element to it; it's purely about learning that the walls only block the player because they are STOP. Nor would (from my experience watching streamers at least) a human spontaneously notice that the rule "WALL IS WIN" had been formed and treat that as a cue to reconsider the entire strategy. The natural human response to unintentionally forming a useful rule is to keep pushing in the same direction. On the other hand, an actually dedicated AI system (in the way that AlphaGo was dedicated to Go) could, I'm sure, figure out a game like Baba Is You pretty easily. It would lack the human instinct to treat the walls as if they were implicitly always STOP; so it would never struggle with overriding it.

deadbabe 1y ago
A simple feed-forward neural network with sufficient training can solve levels way better than Claude. Why is Claude being used at all.

dezgeg 1y ago

Cool, it's something I've been trying also. Here are my attempts: https://www.youtube.com/watch?v=JZ4-YUuCVNo&list=PL8C_UWcLmvGlv7dWY-oSowUfgFnDOeDg2 https://www.youtube.com/watch?v=JZ4-YUuCVNo&list=PL8C_UWcLmv...