4 ms·

On one hand, I don’t really understand why anyone would expect an LLM to solve logic puzzles. The only way it can do so is not through reasoning, but by having

On one hand, I don’t really understand why anyone would expect an LLM to solve logic puzzles. The only way it can do so is not through reasoning, but by having been trained on a structurally similar puzzle. On the other hand, it does feel fun that the top ones appear to solve it, and I understand why it feels cool to have a computer that appears to be capable of solving these puzzles. But really, I think this is just specificity in training. There is no theoretical or empirical basis for LLMs having any reasoning capability. The only reason it can solve it is because side the creators of these top models specifically trained the models on problems like this to give the appearance of intelligence.

mg 3y ago
There might be no reasoning in a single pass which outputs a single token. But in the loop where the output of the LLM repeatedly gets fed back into its input, reasoning is clearly happening: The LLMs lay out how to go about figuring out the answer, do a series of calculation steps and then come up with an answer. If you add "Please answer in just one short sentence." to the prompt, even the top ones get it wrong.
- visarga 3y ago
  Reasoning is also an iterative process. Besides scaling in response length, the model can also get multiple feedbacks from outside to correct itself.
- spyder 3y ago
  Yep, humans too have to think before answering most non-trivial questions, and especially the ones that include calculations. So it seems "obvious" that we should try to to give LLMs too some time to think before answering, for example with the popular methods of asking for step-by-step thinking, thinking out loud, and only giving the final answer at the end, and also asking it to proofread and correct it's answers at the end all can help with that. Pause tokens (thinking tokens) are also an interesting method to achieve that and seems to have a positive effect on performance: https://arxiv.org/abs/2310.02226 https://arxiv.org/abs/2310.02226
visarga 3y ago
> There is no theoretical or empirical basis for LLMs having any reasoning capability. Yes there is. Learning to predict the next token implies a lot of things, among which is also logical reasoning. The chain-of-thought approach shows that when you stimulate this behavior, you get higher accuracies.
xcv123 3y ago
> There is no theoretical or empirical basis for LLMs having any reasoning capability. Deep learning models are specifically designed for automatic pattern recognition. That includes patterns of reasoning and problem solving. > The only reason it can solve it is because side the creators of these top models specifically trained the models on problems like this to give the appearance of intelligence. That's not how deep learning works, and not how machine learning works in general. The models can automatically recognize patterns of reasoning then apply those methods to problems it has never seen before. > The only way it can do so is not through reasoning, but by having been trained on a structurally similar puzzle. This is a fundamental misunderstanding of how it works. The large deep learning models have 100+ layers, modelling extremely abstract features of the data, which include abstract patterns of problem solving and reasoning. They are not simply regurgitating training examples.
xcv123 3y ago
> There is no theoretical or empirical basis for LLMs having any reasoning capability. Geoffrey Hinton - Mapping Part-Whole Hierarchies into Connectionist Networks (1990) https://www.cs.toronto.edu/~hinton/absps/AIJmapping.pdf https://www.cs.toronto.edu/~hinton/absps/AIJmapping.pdf "The paper, titled "Mapping Part-Whole Hierarchies into Connectionist Networks" (1990), demonstrated how neural networks can learn to represent conceptual hierarchies and reason about relations like family trees. Specifically, Hinton showed that by training a neural network on examples of family relationships (parent-child, grandparent-grandchild, etc.), the network was able to accurately model the inherent logical patterns and reason about new family tree instances it had not encountered during training. This pioneering work highlighted that instead of just memorizing specific training examples, neural networks can extract the underlying logical rules and reasoning patterns governing the data. The learned representations captured abstract concepts like "parent" that enabled generalizing to reason about entirely new family tree configurations."
stevenhuang 3y ago
Your assertion that LLMs cannot reason is some exquisite irony considering the extensive theoretical foundation in support of the idea.

mg 3y ago

There might be no reasoning in a single pass which outputs a single token. But in the loop where the output of the LLM repeatedly gets fed back into its input, reasoning is clearly happening: The LLMs lay out how to go about figuring out the answer, do a series of calculation steps and then come up with an answer. If you add "Please answer in just one short sentence." to the prompt, even the top ones get it wrong.

visarga 3y ago
Reasoning is also an iterative process. Besides scaling in response length, the model can also get multiple feedbacks from outside to correct itself.
spyder 3y ago
Yep, humans too have to think before answering most non-trivial questions, and especially the ones that include calculations. So it seems "obvious" that we should try to to give LLMs too some time to think before answering, for example with the popular methods of asking for step-by-step thinking, thinking out loud, and only giving the final answer at the end, and also asking it to proofread and correct it's answers at the end all can help with that. Pause tokens (thinking tokens) are also an interesting method to achieve that and seems to have a positive effect on performance: https://arxiv.org/abs/2310.02226 https://arxiv.org/abs/2310.02226

visarga 3y ago

> There is no theoretical or empirical basis for LLMs having any reasoning capability. Yes there is. Learning to predict the next token implies a lot of things, among which is also logical reasoning. The chain-of-thought approach shows that when you stimulate this behavior, you get higher accuracies.

xcv123 3y ago

> There is no theoretical or empirical basis for LLMs having any reasoning capability. Deep learning models are specifically designed for automatic pattern recognition. That includes patterns of reasoning and problem solving. > The only reason it can solve it is because side the creators of these top models specifically trained the models on problems like this to give the appearance of intelligence. That's not how deep learning works, and not how machine learning works in general. The models can automatically recognize patterns of reasoning then apply those methods to problems it has never seen before. > The only way it can do so is not through reasoning, but by having been trained on a structurally similar puzzle. This is a fundamental misunderstanding of how it works. The large deep learning models have 100+ layers, modelling extremely abstract features of the data, which include abstract patterns of problem solving and reasoning. They are not simply regurgitating training examples.

xcv123 3y ago

> There is no theoretical or empirical basis for LLMs having any reasoning capability. Geoffrey Hinton - Mapping Part-Whole Hierarchies into Connectionist Networks (1990) https://www.cs.toronto.edu/~hinton/absps/AIJmapping.pdf https://www.cs.toronto.edu/~hinton/absps/AIJmapping.pdf "The paper, titled "Mapping Part-Whole Hierarchies into Connectionist Networks" (1990), demonstrated how neural networks can learn to represent conceptual hierarchies and reason about relations like family trees. Specifically, Hinton showed that by training a neural network on examples of family relationships (parent-child, grandparent-grandchild, etc.), the network was able to accurately model the inherent logical patterns and reason about new family tree instances it had not encountered during training. This pioneering work highlighted that instead of just memorizing specific training examples, neural networks can extract the underlying logical rules and reasoning patterns governing the data. The learned representations captured abstract concepts like "parent" that enabled generalizing to reason about entirely new family tree configurations."

stevenhuang 3y ago

Your assertion that LLMs cannot reason is some exquisite irony considering the extensive theoretical foundation in support of the idea.