6 ms·

> I had someone much knowledgable on this topic than myself claim ChatGPT and the like "understand" stuff. My standing argument is that they wouldn't hallucinat

> I had someone much knowledgable on this topic than myself claim ChatGPT and the like "understand" stuff. My standing argument is that they wouldn't hallucinate incorrect responses if they understood anything, the hallucination is when their approximation of what a real response would be falls short. ChatGPT models semantic relationships in the data. That's what your smart buddy means by "understanding". That is a high dimensional model of the data set which infers abstract semantic relationships / concepts. But he would not claim that those semantic relationships are exactly the same as any human interpretation of the data (which you refer to as the "real" response). Language models also have limited reasoning abilities. They are capable of misunderstanding as much as understanding.

PaulDavisThe1st 3y ago
Not bad, but: > Language models also have limited reasoning abilities. They have no reasoning abilities at all.
- Mike_12345 3y ago
  "Language Models Perform Reasoning via Chain of Thought" Posted by Jason Wei and Denny Zhou, Research Scientists, Google Research, Brain team https://ai.googleblog.com/2022/05/language-models-perform-reasoning-via.html https://ai.googleblog.com/2022/05/language-models-perform-re...
  - PaulDavisThe1st 3y ago
    This paper is misusing the term "reasoning" in my opinion. At no point does the LLM know that 5+6 = 11, and if asked to solve a problem in which 5+6 was an implicit component of the solution but not explicitly present in the text, it would be completely lost.
    - deleted 3y ago
      [deleted]
    - Mike_12345 3y ago
      You have a narrow definition of reasoning. Formally and technically it is solving a symbolic reasoning task through a sequence of steps. Yes we know it's not conscious and not human reasoning. > At no point does the LLM know that 5+6 = 11 Does it need to "know" that (by your narrow definition of "know") in order to reason about a word math problem? > if asked to solve a problem in which 5+6 was an implicit component of the solution but not explicitly present in the text, it would be completely lost Can you provide an example? What makes you believe it can't be trained to solve those too? That's just a higher abstraction over the language. Add more layers, more training, etc. Many humans cannot solve basic math word puzzles that this artificial neural network can already solve.
      - PaulDavisThe1st 3y ago
        They do not "solve" word puzzles. They output text that appears to be the best response to the prompt, based on their training data. If the puzzle is solvable by doing this, then they get the answer right. If the puzzle is not solvable doing that, they are unlikely to get the answer right. If I ask you to multiply two (largeish) numbers together, you will be able to do so, using an algorithm/process that you can apply to the multiplication of any two numbers, whether anyone has ever told you about those numbers before or not. LLM's cannot do this. Give them a math problem that doesn't exist in their training set and they cannot solve it. This has been demonstrated many times.
        Mike_12345 3y ago
        > Give them a math problem that doesn't exist in their training set and they cannot solve it. They routinely solve math problems (and other reasoning tasks) that don't exist in their training set. Examples were in that paper I linked to. This is one of the incredible emergent properties of LLMs / deep neural networks. Try it out today on GPT-4. Make up your own math problems and go for it.
        PaulDavisThe1st 3y ago
        PROMPT: what is 19192920 * 190101271 RESPONSE: The product of 19192920 and 190101271 is: 3653424693064240 Fail on first try.
    - flangola7 3y ago
      What is a reasoning task we could give an LLM that would demonstrate that it actually is not reasoning? It seems like that should be easy to construct as a very simple task outside its training set would fail utterly, but I have yet to witness one.
      - PaulDavisThe1st 3y ago
        1a. generate two numbers using: (random() % BIGNUM) 1b. ask the LLM to multiply them together Any human who has learned multiplication can do this. AFAIU, LLMs cannot unless the computation exists within the training set. They have zero arithmetic reasoning capability. I just asked ChatGPT: "19191920 multipled by 10292111772" It said: -------------- To multiply 19191920 by 10292111772, you can use the standard long multiplication method as follows: 19191920 x 10292111772 ------------- 19191920000 (the product of 19191920 and 1) 153535360000 (the product of 19191920 and 8) 1535353600000 (the product of 19191920 and 2) ------------- 196837644266310240 (the final product) Therefore, the result of multiplying 19191920 by 10292111772 is 196837644266310240. ---------------------------------- This is completely wrong. It is not doing arithmetic, and it is not capable of doing arithmetic.
        Mike_12345 3y ago
        All you have shown here is that its arithmetic reasoning is flawed, not that it cannot reason at all. "Reasoning" is a much broader term than you think it is.
        flangola7 3y ago
        Many humans would not be able to solve that problem, especially those that are younger or have lower IQs, and obviously those that have not not been taught multiplication. That doesn't mean they are devoid of all reasoning ability. Furthermore GPT-4 is able to multiple slightly shorter pairs of numbers, and experiments on smaller neural nets, such as one using a dataset consisting of 20% of all combinations of two four digit numbers, quickly learn to generalize to successfully multiply any four digit pair even though they aren't in the training set. Try again, give an answer that actually works this time.
- visarga 3y ago
  But language itself does, it's not the model, it's the data that has this ability. It's in the language patterns. Humans use that too - 99.99% of what reasoning we do is just replaying older ideas adapted in context. Being truly original and improving on the best ideas is a rare thing.
  - PaulDavisThe1st 3y ago
    > 99.99% of what reasoning we do is just replaying older ideas adapted in context. Asserted without evidence. I'm willing to accept that most human speech behavior may well happen in similar ways to LLMs. But most human speech behavior is not reasoning.
    - visarga 3y ago
      I was thinking about this: https://www.wasyresearch.com/content/images/2021/08/the_essence_of_PhD_2-1.jpg https://www.wasyresearch.com/content/images/2021/08/the_esse...
_8j50 3y ago
When I say understanding I mean the meaning and context around it. It is one thing to know how to respond to different inputs surrounding a context, it is another to undersrand why! Now, for all the disagreements in this thread I have not seen anyone claim it knows and reasons why things are the way they are, I don't disagree that it knows what to do for different inputs but it is not processing the input and making a reasonable decision more like mapping input with most approximate output. Is thay incorrect of me to say?
- Mike_12345 3y ago
  > When I say understanding I mean the meaning and context around it. Yes it is actually modelling the meaning and context around it. ChatGPT is a ~100 layer deep neural network that was trained specifically to solve "natural language understanding tasks". The term "understanding" is an important term in NLP research along with the concept of "semantic similarity". I believe it is more powerful and ultimately (eventually) more dangerous than you fear it is.
  - _8j50 3y ago
    Based on what I have see it is making a really good approximation. Here is a thought problem: if memorized every single result of of a multiplication between two numbers up to the maximum a human can possibly think of, then do I understand multiplication? My point about mistakes is not that they were made but the way they were made indicates it was attempting to approximate. Someone mentioned a famous mathematician who got basic multiplication wrong, now if you are wrong because you missed some steps in the process that can be proven. But if your mistake is because you guessed the that can also proven by showing defects in your answer were arbitrary. Pretend you're grading a student, you can tell when they guessed and got it wrong as opposed to tried to follow the process bur misunderstood something or make a critical error.
    - Mike_12345 3y ago
      > if memorized every single result of of a multiplication between two numbers up to the maximum a human can possibly think of, then do I understand multiplication? No, ChatGPT is not simply regurgitating. It actually understands the data. It is one of the largest deep neural networks ever constructed. Consider AlphaZero, the neural net that learned how to play Go and became the world champion. It started with nothing but the rules of the game. Deep Learning systems build a generalized model and reason about it. They are not simply regurgitating what they have already seen in their training set.

PaulDavisThe1st 3y ago

Not bad, but: > Language models also have limited reasoning abilities. They have no reasoning abilities at all.

Mike_12345 3y ago
"Language Models Perform Reasoning via Chain of Thought" Posted by Jason Wei and Denny Zhou, Research Scientists, Google Research, Brain team https://ai.googleblog.com/2022/05/language-models-perform-reasoning-via.html https://ai.googleblog.com/2022/05/language-models-perform-re...
- PaulDavisThe1st 3y ago
  This paper is misusing the term "reasoning" in my opinion. At no point does the LLM know that 5+6 = 11, and if asked to solve a problem in which 5+6 was an implicit component of the solution but not explicitly present in the text, it would be completely lost.
  - deleted 3y ago
    [deleted]
  - Mike_12345 3y ago
    You have a narrow definition of reasoning. Formally and technically it is solving a symbolic reasoning task through a sequence of steps. Yes we know it's not conscious and not human reasoning. > At no point does the LLM know that 5+6 = 11 Does it need to "know" that (by your narrow definition of "know") in order to reason about a word math problem? > if asked to solve a problem in which 5+6 was an implicit component of the solution but not explicitly present in the text, it would be completely lost Can you provide an example? What makes you believe it can't be trained to solve those too? That's just a higher abstraction over the language. Add more layers, more training, etc. Many humans cannot solve basic math word puzzles that this artificial neural network can already solve.
    - PaulDavisThe1st 3y ago
      They do not "solve" word puzzles. They output text that appears to be the best response to the prompt, based on their training data. If the puzzle is solvable by doing this, then they get the answer right. If the puzzle is not solvable doing that, they are unlikely to get the answer right. If I ask you to multiply two (largeish) numbers together, you will be able to do so, using an algorithm/process that you can apply to the multiplication of any two numbers, whether anyone has ever told you about those numbers before or not. LLM's cannot do this. Give them a math problem that doesn't exist in their training set and they cannot solve it. This has been demonstrated many times.
      - Mike_12345 3y ago
        > Give them a math problem that doesn't exist in their training set and they cannot solve it. They routinely solve math problems (and other reasoning tasks) that don't exist in their training set. Examples were in that paper I linked to. This is one of the incredible emergent properties of LLMs / deep neural networks. Try it out today on GPT-4. Make up your own math problems and go for it.
        PaulDavisThe1st 3y ago
        PROMPT: what is 19192920 * 190101271 RESPONSE: The product of 19192920 and 190101271 is: 3653424693064240 Fail on first try.
  - flangola7 3y ago
    What is a reasoning task we could give an LLM that would demonstrate that it actually is not reasoning? It seems like that should be easy to construct as a very simple task outside its training set would fail utterly, but I have yet to witness one.
    - PaulDavisThe1st 3y ago
      1a. generate two numbers using: (random() % BIGNUM) 1b. ask the LLM to multiply them together Any human who has learned multiplication can do this. AFAIU, LLMs cannot unless the computation exists within the training set. They have zero arithmetic reasoning capability. I just asked ChatGPT: "19191920 multipled by 10292111772" It said: -------------- To multiply 19191920 by 10292111772, you can use the standard long multiplication method as follows: 19191920 x 10292111772 ------------- 19191920000 (the product of 19191920 and 1) 153535360000 (the product of 19191920 and 8) 1535353600000 (the product of 19191920 and 2) ------------- 196837644266310240 (the final product) Therefore, the result of multiplying 19191920 by 10292111772 is 196837644266310240. ---------------------------------- This is completely wrong. It is not doing arithmetic, and it is not capable of doing arithmetic.
      - Mike_12345 3y ago
        All you have shown here is that its arithmetic reasoning is flawed, not that it cannot reason at all. "Reasoning" is a much broader term than you think it is.
      - flangola7 3y ago
        Many humans would not be able to solve that problem, especially those that are younger or have lower IQs, and obviously those that have not not been taught multiplication. That doesn't mean they are devoid of all reasoning ability. Furthermore GPT-4 is able to multiple slightly shorter pairs of numbers, and experiments on smaller neural nets, such as one using a dataset consisting of 20% of all combinations of two four digit numbers, quickly learn to generalize to successfully multiply any four digit pair even though they aren't in the training set. Try again, give an answer that actually works this time.
visarga 3y ago
But language itself does, it's not the model, it's the data that has this ability. It's in the language patterns. Humans use that too - 99.99% of what reasoning we do is just replaying older ideas adapted in context. Being truly original and improving on the best ideas is a rare thing.
- PaulDavisThe1st 3y ago
  > 99.99% of what reasoning we do is just replaying older ideas adapted in context. Asserted without evidence. I'm willing to accept that most human speech behavior may well happen in similar ways to LLMs. But most human speech behavior is not reasoning.
  - visarga 3y ago
    I was thinking about this: https://www.wasyresearch.com/content/images/2021/08/the_essence_of_PhD_2-1.jpg https://www.wasyresearch.com/content/images/2021/08/the_esse...

_8j50 3y ago

When I say understanding I mean the meaning and context around it. It is one thing to know how to respond to different inputs surrounding a context, it is another to undersrand why! Now, for all the disagreements in this thread I have not seen anyone claim it knows and reasons why things are the way they are, I don't disagree that it knows what to do for different inputs but it is not processing the input and making a reasonable decision more like mapping input with most approximate output. Is thay incorrect of me to say?

Mike_12345 3y ago
> When I say understanding I mean the meaning and context around it. Yes it is actually modelling the meaning and context around it. ChatGPT is a ~100 layer deep neural network that was trained specifically to solve "natural language understanding tasks". The term "understanding" is an important term in NLP research along with the concept of "semantic similarity". I believe it is more powerful and ultimately (eventually) more dangerous than you fear it is.
- _8j50 3y ago
  Based on what I have see it is making a really good approximation. Here is a thought problem: if memorized every single result of of a multiplication between two numbers up to the maximum a human can possibly think of, then do I understand multiplication? My point about mistakes is not that they were made but the way they were made indicates it was attempting to approximate. Someone mentioned a famous mathematician who got basic multiplication wrong, now if you are wrong because you missed some steps in the process that can be proven. But if your mistake is because you guessed the that can also proven by showing defects in your answer were arbitrary. Pretend you're grading a student, you can tell when they guessed and got it wrong as opposed to tried to follow the process bur misunderstood something or make a critical error.
  - Mike_12345 3y ago
    > if memorized every single result of of a multiplication between two numbers up to the maximum a human can possibly think of, then do I understand multiplication? No, ChatGPT is not simply regurgitating. It actually understands the data. It is one of the largest deep neural networks ever constructed. Consider AlphaZero, the neural net that learned how to play Go and became the world champion. It started with nothing but the rules of the game. Deep Learning systems build a generalized model and reason about it. They are not simply regurgitating what they have already seen in their training set.