5 ms·

LLMs can not plan. There is no LLM that can solve sudoku puzzles by executing the obvious constraint propagation algorithm with backtracking. Therefore, LLMs ca

LLMs can not plan. There is no LLM that can solve sudoku puzzles by executing the obvious constraint propagation algorithm with backtracking. Therefore, LLMs can neither reason nor plan. Software is not magic and the fact that a lot of people are starting to think that it is should be concerning for the folks training the next generation of software engineers.

sublinear 3y ago
I would hope the next generation of softwares engineers are not the kind to blindly buy into media hype.
- dkersten 3y ago
  The next generation (going by some comments I’ve read on HN) are already non-programmers gluing (I assume terrible) LLM-generated code together until it (appears like it) works. Hell, my non-programmer brother recently sent me a message like “can you think of a way to fix this script, ChatGPT isn’t managing” (sends me a badly written AI-generated script)
  - ponderings 3y ago
    > “can you think of a way to fix this script, ChatGPT isn’t managing” haha, who's the glue coder now! edit: I chat with a 13 year old glue coder one time. He couldn't really code at all but he could find the components and was excellent at prompting people for help. He showed many things like map api's interacting with many other things. How long did that take you to put together? uhh 2 hours. My mind was blown.
jasfi 3y ago
What about an LLM writing an algorithm than can play Sudoku? We build things that we ourselves can't do all the time. Edit: I updated my sentence to "an LLM writing an algorithm" to make myself clearer. After reading my own sentence I realized it wasn't, sorry!
- wildrhythms 3y ago
  Of course we can write an algorithm to solve Sudoku. The commenter specifically is talking about LLMs, not algorithms in general.
  - jasfi 3y ago
    I'm making the point that they're not as limited as they seem at first glance.
- jbergens 3y ago
  Did you read the article? The main issue with your idea is that an LLM won't know if the algorithm it created is any good, or even if it works at all. If it can't check that it will never know and never get better. You could ask it to generate a number of algorithms and then yourself choose the best one but then you have worked as a team, the LLM did not plan anything.
  - reaperman 3y ago
    LLM's can integrate with a sandbox to deploy and test their code and iterate on a solution until it appears to be valid. They can also integrate with web search to go out and find presumably-valid sudoku puzzles to use for test cases if they're (likely) unable to generate valid sudoku puzzles themselves. I know it's expanding the definition of "LLM" to include a sandbox or web search, but I think it's fair because it's a reasonably practical and obvious application environment for an LLM which you plan to ask to do things like this, and I think LLMs with both these integrations will be commonplace in the next 1-2 years. No, I don't think LLM's can "reason and plan". But I do think they can effectively mimic (fake) "reasoning and planning" and still arrive at the same result that actual reasoning and planning would yield, for reasonably common and problems of greater than trivial complexity but less than moderate complexity. I think pretty much all of our production AI models today are limited by their lack of ability to self-assess and "goal-seek" and mutate themselves themselves to "excel". I'm not 100% sure what this would look like but I can be sure they don't have any real "drive to excel beyond". Perhaps improvements in Reinforcement Learning will uncover something like this, but I think there may need to be a paradigm shift before we invent something like that.
Drakim 3y ago
I played asci Tic-Tac-Toe with ChatGPT using new rules that I came up with, and it was able to play using the new rules and even explained it's reasoning for the moves when I asked. I have a hard time understanding that as not including thinking and planning.
- abrichr 3y ago
  Sounds interesting! Can you please share the transcript?
  - Drakim 3y ago
    I'm looking for it, but unfortunately I can't find it, which is weird because I've never deliberately deleted any of the transcripts. The homemade rules I used were as follows (each for a separate round): 1. The first player to get three in a row loses. 2. The first player to take the center square loses. 3. Players can place their marker in squares already taken. Admittedly it didn't nail it perfectly, a few times it messed up, but most of the time it was able to follow the rules as explained without issue, and could explain it's reasoning when I asked it why it picked that square.
- intended 3y ago
  This is the call out though - the difference between a proof of concept, and then seeing what happens in production. Production and scale definitively shows that LLMs don't reason. The output is highly unpredictable. Small changes can result in absolutely unrelated outcomes. If you were to classify text using chat gpt - chat gpt switches from classification to text generation if your text is longer than a certain length. LLM based Agents are the place where LLM reasoning died in practice. I really hope that there is a change, maybe larger context windows will allow the generated text to "reason". Although, that would not necessarily be reasoning. Even the article linked, "Indeed, LLMs make it easy to get problem-specific knowledge as long as we are willing to relax correctness requirements of that knowledge." LLMs generate text that approximates what a plan would "sound" like. It doesn't plan.
  - deleted 3y ago
circuit10 3y ago
Would an average human be able to do that on a strict time limit (which LLMs effectively have as they do a fixed amount of computation per token)?
- appplication 3y ago
  The mark for reasoning in a computer program is not whether or not a human can do so on the same time limit. A calculator doesn’t reason but solves 658236 x 37854285 faster than anything human could. You could give an LLM days per token and it wouldn’t change its capabilities regarding reasoning.
  - circuit10 3y ago
    I'm not saying that this is proof that it can reason, but rather that the fact that it can't isn't proof that it can't on its own
famouswaffles 3y ago
Yes LLMs can solve sudoku https://arxiv.org/abs/2305.08291 https://arxiv.org/abs/2305.08291
- gamblor956 3y ago
  No, the LLM did not solve sudoku. They simply linked a sudoku solver to an LLM. All the LLM did was constantly invoke the solver. The solver can be downloaded from github, and works independently of the LLM. All that experiment demonstrated is that you can use an LLM to run through a state tree...which is something that simple machine agents have been able to do for a few decades.
  - famouswaffles 3y ago
    >They simply linked a sudoku solver to an LLM. All the LLM did was constantly invoke the solver. No they didn't just link a solver. The LLM does the filling. It makes an attempt, the checker checks if it's a valid move for sudoku, If not then returns to the previous state(node) and so on until solved. The history of attempts is stored and retrieved every time the LLM backtracks
Buttons840 3y ago
Do 4 year old children plan? They can't solve Sudoku, so arguably not.
- haltist 3y ago
  Children have no explict conceptions of numbers and logic so they obviously do not plan the same way someone with explicit knowledge and understanding of logic reasons and plans. One could argue they have implicit understanding as members of a species known for inventing mathematics but that's more of a philosophical argument than a scientific one.
theonlybutlet 3y ago
Of course it can given the correct framework. I think that's more a limitation of the structure of its interface and programming around it. Allow it to consider and iterate on its own output and it'll get there.

sublinear 3y ago

I would hope the next generation of softwares engineers are not the kind to blindly buy into media hype.

dkersten 3y ago
The next generation (going by some comments I’ve read on HN) are already non-programmers gluing (I assume terrible) LLM-generated code together until it (appears like it) works. Hell, my non-programmer brother recently sent me a message like “can you think of a way to fix this script, ChatGPT isn’t managing” (sends me a badly written AI-generated script)
- ponderings 3y ago
  > “can you think of a way to fix this script, ChatGPT isn’t managing” haha, who's the glue coder now! edit: I chat with a 13 year old glue coder one time. He couldn't really code at all but he could find the components and was excellent at prompting people for help. He showed many things like map api's interacting with many other things. How long did that take you to put together? uhh 2 hours. My mind was blown.

jasfi 3y ago

What about an LLM writing an algorithm than can play Sudoku? We build things that we ourselves can't do all the time. Edit: I updated my sentence to "an LLM writing an algorithm" to make myself clearer. After reading my own sentence I realized it wasn't, sorry!

wildrhythms 3y ago
Of course we can write an algorithm to solve Sudoku. The commenter specifically is talking about LLMs, not algorithms in general.
- jasfi 3y ago
  I'm making the point that they're not as limited as they seem at first glance.
jbergens 3y ago
Did you read the article? The main issue with your idea is that an LLM won't know if the algorithm it created is any good, or even if it works at all. If it can't check that it will never know and never get better. You could ask it to generate a number of algorithms and then yourself choose the best one but then you have worked as a team, the LLM did not plan anything.
- reaperman 3y ago
  LLM's can integrate with a sandbox to deploy and test their code and iterate on a solution until it appears to be valid. They can also integrate with web search to go out and find presumably-valid sudoku puzzles to use for test cases if they're (likely) unable to generate valid sudoku puzzles themselves. I know it's expanding the definition of "LLM" to include a sandbox or web search, but I think it's fair because it's a reasonably practical and obvious application environment for an LLM which you plan to ask to do things like this, and I think LLMs with both these integrations will be commonplace in the next 1-2 years. No, I don't think LLM's can "reason and plan". But I do think they can effectively mimic (fake) "reasoning and planning" and still arrive at the same result that actual reasoning and planning would yield, for reasonably common and problems of greater than trivial complexity but less than moderate complexity. I think pretty much all of our production AI models today are limited by their lack of ability to self-assess and "goal-seek" and mutate themselves themselves to "excel". I'm not 100% sure what this would look like but I can be sure they don't have any real "drive to excel beyond". Perhaps improvements in Reinforcement Learning will uncover something like this, but I think there may need to be a paradigm shift before we invent something like that.

Drakim 3y ago

I played asci Tic-Tac-Toe with ChatGPT using new rules that I came up with, and it was able to play using the new rules and even explained it's reasoning for the moves when I asked. I have a hard time understanding that as not including thinking and planning.

abrichr 3y ago
Sounds interesting! Can you please share the transcript?
- Drakim 3y ago
  I'm looking for it, but unfortunately I can't find it, which is weird because I've never deliberately deleted any of the transcripts. The homemade rules I used were as follows (each for a separate round): 1. The first player to get three in a row loses. 2. The first player to take the center square loses. 3. Players can place their marker in squares already taken. Admittedly it didn't nail it perfectly, a few times it messed up, but most of the time it was able to follow the rules as explained without issue, and could explain it's reasoning when I asked it why it picked that square.
intended 3y ago
This is the call out though - the difference between a proof of concept, and then seeing what happens in production. Production and scale definitively shows that LLMs don't reason. The output is highly unpredictable. Small changes can result in absolutely unrelated outcomes. If you were to classify text using chat gpt - chat gpt switches from classification to text generation if your text is longer than a certain length. LLM based Agents are the place where LLM reasoning died in practice. I really hope that there is a change, maybe larger context windows will allow the generated text to "reason". Although, that would not necessarily be reasoning. Even the article linked, "Indeed, LLMs make it easy to get problem-specific knowledge as long as we are willing to relax correctness requirements of that knowledge." LLMs generate text that approximates what a plan would "sound" like. It doesn't plan.
- deleted 3y ago

circuit10 3y ago

Would an average human be able to do that on a strict time limit (which LLMs effectively have as they do a fixed amount of computation per token)?

appplication 3y ago
The mark for reasoning in a computer program is not whether or not a human can do so on the same time limit. A calculator doesn’t reason but solves 658236 x 37854285 faster than anything human could. You could give an LLM days per token and it wouldn’t change its capabilities regarding reasoning.
- circuit10 3y ago
  I'm not saying that this is proof that it can reason, but rather that the fact that it can't isn't proof that it can't on its own

famouswaffles 3y ago

Yes LLMs can solve sudoku https://arxiv.org/abs/2305.08291 https://arxiv.org/abs/2305.08291

gamblor956 3y ago
No, the LLM did not solve sudoku. They simply linked a sudoku solver to an LLM. All the LLM did was constantly invoke the solver. The solver can be downloaded from github, and works independently of the LLM. All that experiment demonstrated is that you can use an LLM to run through a state tree...which is something that simple machine agents have been able to do for a few decades.
- famouswaffles 3y ago
  >They simply linked a sudoku solver to an LLM. All the LLM did was constantly invoke the solver. No they didn't just link a solver. The LLM does the filling. It makes an attempt, the checker checks if it's a valid move for sudoku, If not then returns to the previous state(node) and so on until solved. The history of attempts is stored and retrieved every time the LLM backtracks

Buttons840 3y ago

Do 4 year old children plan? They can't solve Sudoku, so arguably not.

haltist 3y ago
Children have no explict conceptions of numbers and logic so they obviously do not plan the same way someone with explicit knowledge and understanding of logic reasons and plans. One could argue they have implicit understanding as members of a species known for inventing mathematics but that's more of a philosophical argument than a scientific one.

theonlybutlet 3y ago

Of course it can given the correct framework. I think that's more a limitation of the structure of its interface and programming around it. Allow it to consider and iterate on its own output and it'll get there.