4 ms·

> ChatGPT will gleefully suggest you a wrong approach, just because it’s so intent on satisfying your whim instead of saying No when it would be appropriate Th

> ChatGPT will gleefully suggest you a wrong approach, just because it’s so intent on satisfying your whim instead of saying No when it would be appropriate Therein lies the mistake. Too many people assume ChatGPT (and similar LLMs) are capable of reasoning. It's not. It is simply just giving you what is likely the 'correct' answer based on some sort of pattern. It doesn't know what's wrong, so it's not aware it's giving you an inappropriate answer.

logicallee 2y ago
>Too many people assume ChatGPT (and similar LLMs) are capable of reasoning. It's not. Sure it is, just like a child or someone not very good at reasoning. You can test ChatGPT yourself on some totally novel ad hoc reasoning task you invent for the task, with a single correct conclusion that takes reasoning to arrive at and it will probably get it if it's really easy, even if you take great pains to make it something totally new that you invented. Try it yourself (preferably with ChatGPT 4o) if you don't believe me. Please share your results.
- TeMPOraL 2y ago
  > Sure it is, just like a child or someone not very good at reasoning. That's a good way to think about it. Treat GPT-4 as having mentality of a 4 year old kid. A kid this age will take any question you ask at face value, because it hasn't learned yet that adults often don't ask questions precisely enough, don't realize the assumptions they make in their requests, don't know what they don't know, and are full of shit. A four year old won't think of evaluating whether or not the question itself makes sense, they'll just do their best to answer it, which may involve plain guessing what the answer could be if one isn't apparent. Remember that saying "I don't know" isn't an innate skill in humans either - it's an ability we drill into kids for the first decade or two of their lives.
  - 9dev 2y ago
    That doesn't tell the whole story tho. It's a 4 year old kid that has been thoroughly conditioned to always be positive and affirming in their reply, even if it means making something up. That isn't something kids do usually—it's not something humans usually do, at least not the way ChatGPT does–and that may be part of why it's so confounding. It's not just "I don't know", really. It feels like OpenAI ingrained the essence of North American culture into the model (Sorry North Americans, I really don't mean this in a demeaning way!), as in, the primary task of ChatGPT is supposed to be to make its users happy and feel good about themselves, taking priority over providing accurate answers and facts.
  - Gormo 2y ago
    Another similarity is that 4-year-olds will often pick up words and phrases from people around them, and learn associations between those words and phrases without yet having learned their meanings or having any of their own experiences to relate them to. So a young child might answer your question with a response that he heard other people say to a similar question, without actually understanding what he's saying. LLMs are basically this at a grand scale.
- Gormo 2y ago
  A child or unintelligent person may make errors when attempting to engage in reasoning -- applying the wrong deductive rules, or applying the right ones incorrectly, or applying them to invalid premises -- but LLMs are not even attempting to engage in reasoning in the first place. They are applying no deductive rules, and have no semantic awareness of any of their inputs, so cannot even determine whether they are correct or incorrect. LLMs are simply making probabilistic inferences about what "words" (tokenized particles of language, not necessarily individual words from our perspetive) are most likely to appear in relation to other words, based on the training data fed into them. Their output often resembles reasoning simply because the training data contains large amounts of explicit reasoning in it. But there is no actual reasoning process going on in response to your prompt, just probabilistic inference about what words are most closely correlated with the words in your prompt. If you're getting what appear to be reasoned responses to "novel" prompts, then one of two things is likely happening: either (a) your prompt isn't as novel or unique as you thought it was, and the model's probabilistic inference was sufficient to generate a valid response without reasoning, or (b) the response isn't as well-reasoned as it appears, and you are failing to notice its errors. If you want to genuinely test an LLM's ability to engage in reasoning, try throwing a complex math problem at it, or a logic puzzle that trips most people up.
- xdavidliu 2y ago
  As you are making the point, can you please provide the example? If it's really new, posting it as a comment here is not likely to affect LLM training, at least not for a day or so. And even if it is, you can still provide the examples that you are thinking of?
  - logicallee 2y ago
    You can come up with all the examples you want. Make up your own and try it. Here is an example as you asked for, I made it up: https://chatgpt.com/share/211da88c-d97c-4f18-9984-bed783d0f254 https://chatgpt.com/share/211da88c-d97c-4f18-9984-bed783d0f2... Just make up your own.

logicallee 2y ago

>Too many people assume ChatGPT (and similar LLMs) are capable of reasoning. It's not. Sure it is, just like a child or someone not very good at reasoning. You can test ChatGPT yourself on some totally novel ad hoc reasoning task you invent for the task, with a single correct conclusion that takes reasoning to arrive at and it will probably get it if it's really easy, even if you take great pains to make it something totally new that you invented. Try it yourself (preferably with ChatGPT 4o) if you don't believe me. Please share your results.

TeMPOraL 2y ago
> Sure it is, just like a child or someone not very good at reasoning. That's a good way to think about it. Treat GPT-4 as having mentality of a 4 year old kid. A kid this age will take any question you ask at face value, because it hasn't learned yet that adults often don't ask questions precisely enough, don't realize the assumptions they make in their requests, don't know what they don't know, and are full of shit. A four year old won't think of evaluating whether or not the question itself makes sense, they'll just do their best to answer it, which may involve plain guessing what the answer could be if one isn't apparent. Remember that saying "I don't know" isn't an innate skill in humans either - it's an ability we drill into kids for the first decade or two of their lives.
- 9dev 2y ago
  That doesn't tell the whole story tho. It's a 4 year old kid that has been thoroughly conditioned to always be positive and affirming in their reply, even if it means making something up. That isn't something kids do usually—it's not something humans usually do, at least not the way ChatGPT does–and that may be part of why it's so confounding. It's not just "I don't know", really. It feels like OpenAI ingrained the essence of North American culture into the model (Sorry North Americans, I really don't mean this in a demeaning way!), as in, the primary task of ChatGPT is supposed to be to make its users happy and feel good about themselves, taking priority over providing accurate answers and facts.
- Gormo 2y ago
  Another similarity is that 4-year-olds will often pick up words and phrases from people around them, and learn associations between those words and phrases without yet having learned their meanings or having any of their own experiences to relate them to. So a young child might answer your question with a response that he heard other people say to a similar question, without actually understanding what he's saying. LLMs are basically this at a grand scale.
Gormo 2y ago
A child or unintelligent person may make errors when attempting to engage in reasoning -- applying the wrong deductive rules, or applying the right ones incorrectly, or applying them to invalid premises -- but LLMs are not even attempting to engage in reasoning in the first place. They are applying no deductive rules, and have no semantic awareness of any of their inputs, so cannot even determine whether they are correct or incorrect. LLMs are simply making probabilistic inferences about what "words" (tokenized particles of language, not necessarily individual words from our perspetive) are most likely to appear in relation to other words, based on the training data fed into them. Their output often resembles reasoning simply because the training data contains large amounts of explicit reasoning in it. But there is no actual reasoning process going on in response to your prompt, just probabilistic inference about what words are most closely correlated with the words in your prompt. If you're getting what appear to be reasoned responses to "novel" prompts, then one of two things is likely happening: either (a) your prompt isn't as novel or unique as you thought it was, and the model's probabilistic inference was sufficient to generate a valid response without reasoning, or (b) the response isn't as well-reasoned as it appears, and you are failing to notice its errors. If you want to genuinely test an LLM's ability to engage in reasoning, try throwing a complex math problem at it, or a logic puzzle that trips most people up.
xdavidliu 2y ago
As you are making the point, can you please provide the example? If it's really new, posting it as a comment here is not likely to affect LLM training, at least not for a day or so. And even if it is, you can still provide the examples that you are thinking of?
- logicallee 2y ago
  You can come up with all the examples you want. Make up your own and try it. Here is an example as you asked for, I made it up: https://chatgpt.com/share/211da88c-d97c-4f18-9984-bed783d0f254 https://chatgpt.com/share/211da88c-d97c-4f18-9984-bed783d0f2... Just make up your own.