3 ms·
In the spirit of ARC-AGI-3-like challenges, we just tested if frontier AI models are able to solve a lovely puzzle game, Baba Is You: https://quesma.com/blog/ba
by stared 3mo ago
In the spirit of ARC-AGI-3-like challenges, we just tested if frontier AI models are able to solve a lovely puzzle game, Baba Is You: https://quesma.com/blog/baba-is-bench/ https://quesma.com/blog/baba-is-bench/
A year ago, Sonnet 4 barely solved the first level. Now, both Fable 5 and GPT-5.6 Sol beat the first two stages. GPT 5.2 is slow, but efficient, while Gemini 3.1 Pro and 3.5 Flash struggle.
- sva_ 3mo agoI'm wondering what's up with the release of Gemini 3.5 Pro, they keep postponing it. For a while, Google was doing pretty well with their releases.
- pixl97 3mo agoHeh, probably something like this. Works fast - tells people how to overthrow the government. Follows all rules and conventions Google wants - says corporate speak without actually accomplishing anything. Can actually do complicated things- apt to tell the user to fuck off and do the hard work themselves. Training models seems more akin to raising a kid than a computer application.
- striking 3mo agohttps://www.bloomberg.com/news/articles/2026-07-16/google-gemini-launch-delayed-as-tech-falls-short-of-internal-goals https://www.bloomberg.com/news/articles/2026-07-16/google-ge...
- Tadpole9181 3mo agoFWIW: "Baba Is You" is 7 years old and heralded as one of the greatest puzzle games of all times, with guides and solutions shared all over the internet. How to beat this game is 100% in the training set.
- stared 3mo agoIt was our original assumption. Yet, we went through trajectories and agents did not recall solution. It is with a sharp contrast with task for which agents magically generate solution, e.g. https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/ https://openai.com/index/why-we-no-longer-evaluate-swe-bench.... In a few instances (we covered it in Caveats) Gemini 3.5 Flash "knew" which level it was, but misremembered, and went with a wrong solution.
- modeless 3mo agoIt turns out knowledge and ability are not the same thing. We should test both.
- pants2 3mo agoForget Baba Is You, I want to see an LLM beat Elden Ring
- dezgeg 3mo agoI've been trying this too: https://www.youtube.com/watch?v=brkP58pZ23w&list=PL8C_UWcLmvGlv7dWY-oSowUfgFnDOeDg2 https://www.youtube.com/watch?v=brkP58pZ23w&list=PL8C_UWcLmv... Cool to see others trying Baba Is You too.
- scotty79 3mo agoWith this harness or without?