6 ms·

What does “reason” mean? It seems like it does everything I expect from something that reasons.

famouswaffles 4y ago
People routinely make up their own vague and ill defined meanings of understanding and reasoning to disqualify LLMs. This is necessary because LLMs obviously reason and understand by any evaluation that can be carried out. Seriously just watch. He's not actually going to be able to coherently define his "reasoning" in a way that can be tested.
- DonaldPShimoda 4y ago
  > Seriously just watch. He's not actually going to be able to coherently define his "reasoning" in a way that can be tested. Google gives the following definition of the verb "reason": > think, understand, and form judgments by a process of logic. LLMs do not think, they do not understand, and they do not form judgments. They do not come to their own conclusions. They do not have the physical capability. They are statistical models, nothing more. > LLMs obviously reason and understand by any evaluation that can be carried out. Uh-huh. Sure, Jan.
  - famouswaffles 4y ago
    >LLMs do not think, they do not understand, and they do not form judgments. They do not come to their own conclusions. They do not have the physical capability. They are statistical models, nothing more. "LLMs don't reason because they don't understand" is not the bastion of genius you think it is. It's a circular argument that relies on whatever bespoke interpretations you have cooked up. They don't form judgement or conclusions? Sure looks like they do. So what's the difference ? What is GPT-4 doing then when it correctly looks like it is reasoning and what's the difference between that and "real" understanding or reasoning. Such a huge difference I should be able to test for it. Don't understand how you can tell me what I'm seeing isn't real reasoning but fail to provide a way to empirically determine the difference. >Uh-huh. Sure, Jan. Yeah. https://arxiv.org/abs/2212.09196 https://arxiv.org/abs/2212.09196 Many other evaluations to carry out.
  - kqr 4y ago
    This is the problem with non-operational definitions, because now we need to know how you define "think" and "understand" and "form judgments", to move on. Instead, could you operationally define "reason" in a way that a human is, say, 90 % likely to pass the test and GPT is 10 % likely to do?
    - sanxiyn 4y ago
      Yes, François Chollet released ARC(Abstraction and Reasoning Corpus) benchmark for this in 2019, and the benchmark can be scored automatically. Humans solve 100% of tests and GPTs solve 0% of tests and GPTs made exactly zero progress from 2019 to 2022. https://twitter.com/fchollet/status/1631699463524986880 https://twitter.com/fchollet/status/1631699463524986880 https://github.com/fchollet/ARC https://github.com/fchollet/ARC
      - kqr 4y ago
        Wow, this is a much more interesting answer than I expected. Thank you!
        famouswaffles 4y ago
        It is. Although I'd be careful with the conclusions. The problems are presented in a way that make it difficult to solve. The vision problems will require the equivalent of an artificial visual cortex, something we are seriously lacking in artificial intelligence at the moment. Image to text won't cut it here. For the text there could be tokenizer issues. LLMs don't really have any problem with abstract analogical reasoning https://arxiv.org/abs/2212.09196 https://arxiv.org/abs/2212.09196
      - freilanzer 4y ago
        This seems more like an issue with the tokenizers, no? This doesn't seem more difficult than other problems it solves.
        famouswaffles 4y ago
  - cjbprime 4y ago
    New bar for people claiming LLMs can't reason: invent a specific, testable problem, representable in text, that many humans can solve and LLMs can't, and tell us what it is.
    - Godel_unicode 4y ago
      … and then perform a careful search of books and the whole internet to be sure what you think is novel hasn’t been thoroughly debated somewhere on stackexchange.
      - cjbprime 4y ago
        If it's a stochastic parrot, then merely randomizing proper nouns and filler text should be enough to prevent its abstraction ability. If you're saying that we can't use a problem if any analog of that problem has ever been described, you seem to be arguing more strongly that it is a general intelligence than I am.
        Godel_unicode 4y ago
        I’m saying that people in my circle have been asking what they think are novel questions and getting interesting answers, only to find out that very similar content exists on websites we know are in the training set. That’s not intelligence that’s computers having better memory than humans. Useful, certainly, but hardly skynet.
        cjbprime 4y ago
        I don't think you're being clear about whether the questions were novel. If you discover your question was uncreative, surely e.g. some details, wording, facts, names, or numbers inside the question can be changed to defeat a model that is answering it from memory? If you're saying that it is not possible to change the details enough to avoid the model being able to answer that type of question, I think you are admitting that the model has learned a generalized ability to answer questions of that class, and is not actually using its memory to answer at all. I don't care about whether it learned that generalized ability from seeing examples of the question and answer, which it then deduced an algorithm for and generalized -- that's how most people learn most things.
  - sdenton4 4y ago
    We're survival machines, nothing more... Complex behavior arises from simple systems all the time. You can't prove that these systems don't reason, no matter how loudly thou doth protest.
    - Al0neStar 4y ago
      from the beginning of time people have been overestimating the complexity of things like the human brain and attributing it to magical things (like a creator) far beyond our comprehension but what seems to be happening now is that some people are underestimating it.
graypegg 4y ago
I’ve been imagining “reasoning” as meaning “referring to an abstraction”. The idea being if you wanted to write a script to tell you how far an arrow of a given weight will fly, you’ll probably use some physics simulation (or even just a coefficient to multiply the input weight by) to abstract the arrow in flight. Then you can answer questions around that abstraction. The alternative is having 1000 different results for different kinds of arrows, and averaging out the results for the ones similar to the input arrow. An LLM is working on text tokens, it’s trying to give the most statistically common next token based on everything that’s been fed into it. Does that statistical model abstract the objects and concepts it talks about? Eh? I don’t know
- cjbprime 4y ago
  This is one of those comments that begs for actually using LLMs. Yes, an advanced LLM like GPT-4 will absolutely show evidence of an abstracted world model, correctly answering questions about where person X thinks object Y is, or what relation two people have to each other, or playing a text adventure game while clearly following an internal map it has created for itself.
  - graypegg 4y ago
    So I get that the evidence is that you can do insanely complex things with consistency, like playing a text adventure. You pick up a thing and it stays in your inventory etc, ergo, it’s modelled a character with an inventory in a text adventure. vvv But, coming at this as *respectfully and curiously* as I can here: ^^^ I guess I’m a little more skeptical about attributing magic to something that I know is only working from words as a source. Like if it’s seen NOUN really close to VERB a lot, it can “assume” (read: bias output) that NOUN will VERB in the context it sees it in, that’s an abstraction! But a weak one right? Language is pretty inaccurate. The fact that NLP was a whole booming research field a few years ago tells me that just parsing language consistently is a difficult problem. So, I know I can’t give you any convincing argument there since we’re already mentioning how people are giving untestable definitions of “reasoning” and “abstraction” in this thread. And im no better in that respect here, BUT I guess my best inkling of what feels “off” is a lack of precision, inherent to the medium in which it operates which makes it feel like a true abstraction is impossible in an LLM.
ttpphd 4y ago
When you hear someone speak, are you thinking their thoughts? We are talking about a model of language and our expectations for what the words would look like if a person were using language to reason with. People are confusing one thing (their own interpretation of what they are reading) with another (a statistical model that is being driven to align with certain specific human expectations).

People routinely make up their own vague and ill defined meanings of understanding and reasoning to disqualify LLMs. This is necessary because LLMs obviously reason and understand by any evaluation that can be carried out. Seriously just watch. He's not actually going to be able to coherently define his "reasoning" in a way that can be tested.

DonaldPShimoda 4y ago
> Seriously just watch. He's not actually going to be able to coherently define his "reasoning" in a way that can be tested. Google gives the following definition of the verb "reason": > think, understand, and form judgments by a process of logic. LLMs do not think, they do not understand, and they do not form judgments. They do not come to their own conclusions. They do not have the physical capability. They are statistical models, nothing more. > LLMs obviously reason and understand by any evaluation that can be carried out. Uh-huh. Sure, Jan.
- famouswaffles 4y ago
  >LLMs do not think, they do not understand, and they do not form judgments. They do not come to their own conclusions. They do not have the physical capability. They are statistical models, nothing more. "LLMs don't reason because they don't understand" is not the bastion of genius you think it is. It's a circular argument that relies on whatever bespoke interpretations you have cooked up. They don't form judgement or conclusions? Sure looks like they do. So what's the difference ? What is GPT-4 doing then when it correctly looks like it is reasoning and what's the difference between that and "real" understanding or reasoning. Such a huge difference I should be able to test for it. Don't understand how you can tell me what I'm seeing isn't real reasoning but fail to provide a way to empirically determine the difference. >Uh-huh. Sure, Jan. Yeah. https://arxiv.org/abs/2212.09196 https://arxiv.org/abs/2212.09196 Many other evaluations to carry out.
- kqr 4y ago
  This is the problem with non-operational definitions, because now we need to know how you define "think" and "understand" and "form judgments", to move on. Instead, could you operationally define "reason" in a way that a human is, say, 90 % likely to pass the test and GPT is 10 % likely to do?
  - sanxiyn 4y ago
    Yes, François Chollet released ARC(Abstraction and Reasoning Corpus) benchmark for this in 2019, and the benchmark can be scored automatically. Humans solve 100% of tests and GPTs solve 0% of tests and GPTs made exactly zero progress from 2019 to 2022. https://twitter.com/fchollet/status/1631699463524986880 https://twitter.com/fchollet/status/1631699463524986880 https://github.com/fchollet/ARC https://github.com/fchollet/ARC
    - kqr 4y ago
      Wow, this is a much more interesting answer than I expected. Thank you!
      - famouswaffles 4y ago
        It is. Although I'd be careful with the conclusions. The problems are presented in a way that make it difficult to solve. The vision problems will require the equivalent of an artificial visual cortex, something we are seriously lacking in artificial intelligence at the moment. Image to text won't cut it here. For the text there could be tokenizer issues. LLMs don't really have any problem with abstract analogical reasoning https://arxiv.org/abs/2212.09196 https://arxiv.org/abs/2212.09196
    - freilanzer 4y ago
      This seems more like an issue with the tokenizers, no? This doesn't seem more difficult than other problems it solves.
      - famouswaffles 4y ago
- cjbprime 4y ago
  New bar for people claiming LLMs can't reason: invent a specific, testable problem, representable in text, that many humans can solve and LLMs can't, and tell us what it is.
  - Godel_unicode 4y ago
    … and then perform a careful search of books and the whole internet to be sure what you think is novel hasn’t been thoroughly debated somewhere on stackexchange.
    - cjbprime 4y ago
      If it's a stochastic parrot, then merely randomizing proper nouns and filler text should be enough to prevent its abstraction ability. If you're saying that we can't use a problem if any analog of that problem has ever been described, you seem to be arguing more strongly that it is a general intelligence than I am.
      - Godel_unicode 4y ago
        I’m saying that people in my circle have been asking what they think are novel questions and getting interesting answers, only to find out that very similar content exists on websites we know are in the training set. That’s not intelligence that’s computers having better memory than humans. Useful, certainly, but hardly skynet.
        cjbprime 4y ago
        I don't think you're being clear about whether the questions were novel. If you discover your question was uncreative, surely e.g. some details, wording, facts, names, or numbers inside the question can be changed to defeat a model that is answering it from memory? If you're saying that it is not possible to change the details enough to avoid the model being able to answer that type of question, I think you are admitting that the model has learned a generalized ability to answer questions of that class, and is not actually using its memory to answer at all. I don't care about whether it learned that generalized ability from seeing examples of the question and answer, which it then deduced an algorithm for and generalized -- that's how most people learn most things.
- sdenton4 4y ago
  We're survival machines, nothing more... Complex behavior arises from simple systems all the time. You can't prove that these systems don't reason, no matter how loudly thou doth protest.
  - Al0neStar 4y ago
    from the beginning of time people have been overestimating the complexity of things like the human brain and attributing it to magical things (like a creator) far beyond our comprehension but what seems to be happening now is that some people are underestimating it.

graypegg 4y ago

I’ve been imagining “reasoning” as meaning “referring to an abstraction”. The idea being if you wanted to write a script to tell you how far an arrow of a given weight will fly, you’ll probably use some physics simulation (or even just a coefficient to multiply the input weight by) to abstract the arrow in flight. Then you can answer questions around that abstraction. The alternative is having 1000 different results for different kinds of arrows, and averaging out the results for the ones similar to the input arrow. An LLM is working on text tokens, it’s trying to give the most statistically common next token based on everything that’s been fed into it. Does that statistical model abstract the objects and concepts it talks about? Eh? I don’t know

cjbprime 4y ago
This is one of those comments that begs for actually using LLMs. Yes, an advanced LLM like GPT-4 will absolutely show evidence of an abstracted world model, correctly answering questions about where person X thinks object Y is, or what relation two people have to each other, or playing a text adventure game while clearly following an internal map it has created for itself.
- graypegg 4y ago
  So I get that the evidence is that you can do insanely complex things with consistency, like playing a text adventure. You pick up a thing and it stays in your inventory etc, ergo, it’s modelled a character with an inventory in a text adventure. vvv But, coming at this as *respectfully and curiously* as I can here: ^^^ I guess I’m a little more skeptical about attributing magic to something that I know is only working from words as a source. Like if it’s seen NOUN really close to VERB a lot, it can “assume” (read: bias output) that NOUN will VERB in the context it sees it in, that’s an abstraction! But a weak one right? Language is pretty inaccurate. The fact that NLP was a whole booming research field a few years ago tells me that just parsing language consistently is a difficult problem. So, I know I can’t give you any convincing argument there since we’re already mentioning how people are giving untestable definitions of “reasoning” and “abstraction” in this thread. And im no better in that respect here, BUT I guess my best inkling of what feels “off” is a lack of precision, inherent to the medium in which it operates which makes it feel like a true abstraction is impossible in an LLM.

ttpphd 4y ago

When you hear someone speak, are you thinking their thoughts? We are talking about a model of language and our expectations for what the words would look like if a person were using language to reason with. People are confusing one thing (their own interpretation of what they are reading) with another (a statistical model that is being driven to align with certain specific human expectations).