6 ms·

I just oneshot it with claude code (opus 4.5) using this prompt. It took about 5 mins and included detecting that it was cheating at first (drew a line around t

I just oneshot it with claude code (opus 4.5) using this prompt. It took about 5 mins and included detecting that it was cheating at first (drew a line around the boundary of the maze instead), so it added guardrails for that: ``` Create a devenv project that does the following: - Read the image at maze.jpg - Write a script that solves the maze in the most optimal way between the mouse and the cheese - Generate a new image which is of the original maze, but with a red line that represents the calculated path Use whatever lib/framework is most appropriate ``` Output: https://gist.github.com/J-Swift/ceb1db348f46ba167948f734ff0fc604 Solution: https://imgur.com/a/bkJloPT

esafak 10mo ago
If you allow tool use much simpler models can solve it.
deleted 10mo ago
[deleted]
nl 10mo ago
Programs can solve mazes and LLMs can program. That's a different thing completely.
- JamesSwift 10mo ago
  That just seems like an arbitrary limitation. Its like asking someone to do answer a math calculation but "no thinking allowed". Like, I guess we can gauge if a model just _knows all knowable things in the universe_ using that method... but anything of any value that you are gauging in terms of 'intelligence', is going to actually be validating their ability to go "outside the scope" of what they actually are (an autocomplete on steroids).
  - flyinglizard 10mo ago
    We know there are very simple maze solving algorithms you could code in few lines of Python but no one could claim that constitutes intelligence. The difference is between applying intuitive logic and using a predetermined tool.
  - nearbuy 10mo ago
    It depends whether you're asking it to solve a maze because you just need something that can solve mazes, or if you're trying to learn something about the model's abilities in different domains. If it can't solve a maze by inspection instead of writing a program to solve it, that tells you something about its visual reasoning abilities, and that can help you predict how they'll perform on other visual reasoning tasks that aren't easy to solve with code.
    - seanmcdirmid 10mo ago
      You could actually add mazes and paths through them to the training corpus, or make a model for just solving mazes. I wonder how effective it would be, I’m sure someone has tried it. I doubt it would generalize enough to give the AI new visual reasoning capabilities beyond just solving mazes.
    - JamesSwift 10mo ago
      Again, think about how the models work. They generate text sequentially. Think about how you solve the maze in your mind. Do you draw a line direct to the finish? No, it would be impossible to know what the path was until you had done it. But at that point you have now backtracked several times. So, what could a model _possibly_ be able to do for this puzzle which is "fair game" as a valid solution, other than magically know an answer by pulling it out of thin air?
      - nl 10mo ago
        > So, what could a model _possibly_ be able to do for this puzzle which is "fair game" as a valid solution, other than magically know an answer by pulling it out of thin air? Represent the maze as a sequence of movements which either continue or end up being forced to backtrack. Basically it would represent the maze as a graph and do a depth-first search, keeping track of what nodes it as visited in its reasoning tokens. See for example https://stackoverflow.com/questions/3097556/programming-theory-solve-a-maze https://stackoverflow.com/questions/3097556/programming-theo... where the solution is represented as: A B D (backtrack) E H L (backtrack) M * (backtrack) O (backtrack thrice) I (backtrack thrice) C F (backtrack) G J
  - rglullis 10mo ago
    By your analogy, the developers of stockfish are better chess players than any grandmaster. Tool use can be a sign of intelligence, but "being able to use a tool to solve a problem" is not the same as "being intelligent enough to solve a specific class of problems".
    - JamesSwift 10mo ago
      Im not talking about this being the "best maze solver" and "better at solving mazes than humans". Im saying the model is "intelligent enough" to solve a maze. And what Im really saying is that we need to stop moving the goal post on what "intelligence" is for these models, and start moving the goal post on what "intelligence" actually _is_. The models are giving us an existential crisis on not only what it might mean to _be_ intelligent, but also how it might actually work in our own brains. Im not saying the current models are skynet, but Im saying I think theres going to be a lot learned by reverse engineering the current generation of models to really dig into how they are encoding things internally.
      - rglullis 10mo ago
        > Im saying the model is "intelligent enough" to solve a maze. And I don't agree. I think that at best the model is "intelligent enough to use a tool that can solve mazes" (which is an entirely different thing) and at worst it is no different than a circus horse that "can do math". Being able to repeat more tricks and being able to select which trick to execute based on the expected reward is not a measure of intelligence.
        JamesSwift 10mo ago
        I would encourage you to read the code it produced. Its not like a simple "solve maze" function. There are plenty of "smart" choices in there to achieve the goal given my very vague instructions, and as a result of it analyzing why it failed at first and then adjusting.
        rglullis 10mo ago
sebastiennight 10mo ago
This (writing a program to solve the problem) would be a perfectly valid solution if the model had come up with it. I participated in a "math" competition in high school which mostly tested logic and reasoning. The reason my team won by a landslide is because I showed up with a programmable calculator and knew how to turn the problems into a program that could solve them. By prompting the model to create the program, you're taking away one of the critical reasoning steps needed to solve the problem.

esafak 10mo ago

If you allow tool use much simpler models can solve it.

deleted 10mo ago

[deleted]

nl 10mo ago

Programs can solve mazes and LLMs can program. That's a different thing completely.

JamesSwift 10mo ago
That just seems like an arbitrary limitation. Its like asking someone to do answer a math calculation but "no thinking allowed". Like, I guess we can gauge if a model just _knows all knowable things in the universe_ using that method... but anything of any value that you are gauging in terms of 'intelligence', is going to actually be validating their ability to go "outside the scope" of what they actually are (an autocomplete on steroids).
- flyinglizard 10mo ago
  We know there are very simple maze solving algorithms you could code in few lines of Python but no one could claim that constitutes intelligence. The difference is between applying intuitive logic and using a predetermined tool.
- nearbuy 10mo ago
  It depends whether you're asking it to solve a maze because you just need something that can solve mazes, or if you're trying to learn something about the model's abilities in different domains. If it can't solve a maze by inspection instead of writing a program to solve it, that tells you something about its visual reasoning abilities, and that can help you predict how they'll perform on other visual reasoning tasks that aren't easy to solve with code.
  - seanmcdirmid 10mo ago
    You could actually add mazes and paths through them to the training corpus, or make a model for just solving mazes. I wonder how effective it would be, I’m sure someone has tried it. I doubt it would generalize enough to give the AI new visual reasoning capabilities beyond just solving mazes.
  - JamesSwift 10mo ago
    Again, think about how the models work. They generate text sequentially. Think about how you solve the maze in your mind. Do you draw a line direct to the finish? No, it would be impossible to know what the path was until you had done it. But at that point you have now backtracked several times. So, what could a model _possibly_ be able to do for this puzzle which is "fair game" as a valid solution, other than magically know an answer by pulling it out of thin air?
    - nl 10mo ago
      > So, what could a model _possibly_ be able to do for this puzzle which is "fair game" as a valid solution, other than magically know an answer by pulling it out of thin air? Represent the maze as a sequence of movements which either continue or end up being forced to backtrack. Basically it would represent the maze as a graph and do a depth-first search, keeping track of what nodes it as visited in its reasoning tokens. See for example https://stackoverflow.com/questions/3097556/programming-theory-solve-a-maze https://stackoverflow.com/questions/3097556/programming-theo... where the solution is represented as: A B D (backtrack) E H L (backtrack) M * (backtrack) O (backtrack thrice) I (backtrack thrice) C F (backtrack) G J
- rglullis 10mo ago
  By your analogy, the developers of stockfish are better chess players than any grandmaster. Tool use can be a sign of intelligence, but "being able to use a tool to solve a problem" is not the same as "being intelligent enough to solve a specific class of problems".
  - JamesSwift 10mo ago
    Im not talking about this being the "best maze solver" and "better at solving mazes than humans". Im saying the model is "intelligent enough" to solve a maze. And what Im really saying is that we need to stop moving the goal post on what "intelligence" is for these models, and start moving the goal post on what "intelligence" actually _is_. The models are giving us an existential crisis on not only what it might mean to _be_ intelligent, but also how it might actually work in our own brains. Im not saying the current models are skynet, but Im saying I think theres going to be a lot learned by reverse engineering the current generation of models to really dig into how they are encoding things internally.
    - rglullis 10mo ago
      > Im saying the model is "intelligent enough" to solve a maze. And I don't agree. I think that at best the model is "intelligent enough to use a tool that can solve mazes" (which is an entirely different thing) and at worst it is no different than a circus horse that "can do math". Being able to repeat more tricks and being able to select which trick to execute based on the expected reward is not a measure of intelligence.
      - JamesSwift 10mo ago
        I would encourage you to read the code it produced. Its not like a simple "solve maze" function. There are plenty of "smart" choices in there to achieve the goal given my very vague instructions, and as a result of it analyzing why it failed at first and then adjusting.
        rglullis 10mo ago

sebastiennight 10mo ago

This (writing a program to solve the problem) would be a perfectly valid solution if the model had come up with it. I participated in a "math" competition in high school which mostly tested logic and reasoning. The reason my team won by a landslide is because I showed up with a programmable calculator and knew how to turn the problems into a program that could solve them. By prompting the model to create the program, you're taking away one of the critical reasoning steps needed to solve the problem.