3 ms·

I don’t think they’re mutually exclusive. Next word prediction IS reasoning. It cannot do arbitrarily complex reasoning but many people have used the next word

by twayt 3y ago

I don’t think they’re mutually exclusive. Next word prediction IS reasoning. It cannot do arbitrarily complex reasoning but many people have used the next word prediction mechanism to chain together multiple outputs to produce something akin to reasoning. What definition of reasoning are you operating on?

TacticalCoder 3y ago
> Next word prediction IS reasoning I can write a program in less than 100 lines that can do next work prediction and I guarantee you it's not going to be reasoning. Note that I'm not saying LLMs are or are not reasoning. I'm saying "next word prediction" is not anywhere near sufficient to determine if something is able to reason or not.
- twayt 3y ago
  Any program you write is encoded reasoning. I’d argue if-then statements are reasoning too. Even if you do write a garbage next word predictor, it would still be reasoning. It’s just a qualitative assessment that it would be good reasoning. Again, what exactly is your definition of reasoning? It seems to be not well defined enough to have a discussion about in this context.
agentultra 3y ago
Semantic reasoning, being able to understand what a symbol means and ascertain truth from expressions (which can also mean manipulating expressions in order to derive that truth). As far as I understand tensors and transformers that's... not what they're doing.
- twayt 3y ago
  If you understand transformers, you’d know that they’re doing precisely that. They’re taking a sequence of tokens (symbols), manipulating them (matrix multiplication is ultimately just moving things around and re-weighting - the same operations that you call symbol manipulations can be encoded or at least approximated there) and output a sequence of other tokens (symbols) that make sense to humans. You use the term “ascertain truth” lightly. Unless you’re operating in an axiomatic system or otherwise have access to equipment to query the real world, you can’t really “ascertain truth”. Try using ChatGPT with gpt4 enabled and present it with a novel scenario with well defined rules. That scenario surely isn’t present in its training data but it will able to show signs of making inferences and breaking the problem down. It isn’t just regurgitating memorizing text.
  - agentultra 3y ago
    Oh cool, so we can ask it to give us a proof of the Erdős–Gyárfás conjecture? I’ve seen it confidently regurgitate incorrect proofs of linear algebra theorems. I’m just not confident it’s doing the kind of reasoning needed for us to trust that it can prove theorems formally.
    - Sharlin 3y ago
      Do you also deem humans incapable of reasoning unless they can prove the Erdős–Gyárfás conjecture? Like, talk about moving the goalposts!
    - twayt 3y ago
      Just because it makes mistakes on a domain that may not be part of it's data and/or architectural capabilities doesn't mean it can't do what humans consider "reasoning". Once again, I implore you to come up with a working definition of "reasoning" so that we can have a real discussion about this. Many undergraduates also confidently regurgitate incorrect proofs of linear algebra theorems, do you consider them completely lacking in reasoning ability?
      - agentultra 3y ago
        > Many undergraduates also confidently regurgitate incorrect proofs of linear algebra theorems, do you consider them completely lacking in reasoning ability? No. Because I can ask them questions about their proof, they understand what it means, and can correct it on their own. I've seen LLM's correct their answers after receiving prompts that point out the errors in prior outputs. However I've also seen them give more wrong answers. It tells me that they don't "understand" what it means for an expression to be true or how to derive expressions. For that we'd need some form of deductive reasoning; not generating the next likely token based off a model trained on some input corpus. That's not how most mathematicians seem to do their work. However I think it seems plausible we will have a machine learning algorithm that can do simple inductive proofs and that will be nice. To the original article it seems like they're taking a first step with this. In the mean time why should anyone believe that an LLM is capable of deductive reasoning? Is a tensor enough to represent semantics to be able to dispatch a theorem to an LLM and have it write a proof? Or do I need to train it on enough proofs first before it can start inferring proof-like text?

by twayt 3y ago

TacticalCoder 3y ago

> Next word prediction IS reasoning I can write a program in less than 100 lines that can do next work prediction and I guarantee you it's not going to be reasoning. Note that I'm not saying LLMs are or are not reasoning. I'm saying "next word prediction" is not anywhere near sufficient to determine if something is able to reason or not.

twayt 3y ago
Any program you write is encoded reasoning. I’d argue if-then statements are reasoning too. Even if you do write a garbage next word predictor, it would still be reasoning. It’s just a qualitative assessment that it would be good reasoning. Again, what exactly is your definition of reasoning? It seems to be not well defined enough to have a discussion about in this context.

agentultra 3y ago

Semantic reasoning, being able to understand what a symbol means and ascertain truth from expressions (which can also mean manipulating expressions in order to derive that truth). As far as I understand tensors and transformers that's... not what they're doing.

twayt 3y ago
If you understand transformers, you’d know that they’re doing precisely that. They’re taking a sequence of tokens (symbols), manipulating them (matrix multiplication is ultimately just moving things around and re-weighting - the same operations that you call symbol manipulations can be encoded or at least approximated there) and output a sequence of other tokens (symbols) that make sense to humans. You use the term “ascertain truth” lightly. Unless you’re operating in an axiomatic system or otherwise have access to equipment to query the real world, you can’t really “ascertain truth”. Try using ChatGPT with gpt4 enabled and present it with a novel scenario with well defined rules. That scenario surely isn’t present in its training data but it will able to show signs of making inferences and breaking the problem down. It isn’t just regurgitating memorizing text.
- agentultra 3y ago
  Oh cool, so we can ask it to give us a proof of the Erdős–Gyárfás conjecture? I’ve seen it confidently regurgitate incorrect proofs of linear algebra theorems. I’m just not confident it’s doing the kind of reasoning needed for us to trust that it can prove theorems formally.
  - Sharlin 3y ago
    Do you also deem humans incapable of reasoning unless they can prove the Erdős–Gyárfás conjecture? Like, talk about moving the goalposts!
  - twayt 3y ago
    Just because it makes mistakes on a domain that may not be part of it's data and/or architectural capabilities doesn't mean it can't do what humans consider "reasoning". Once again, I implore you to come up with a working definition of "reasoning" so that we can have a real discussion about this. Many undergraduates also confidently regurgitate incorrect proofs of linear algebra theorems, do you consider them completely lacking in reasoning ability?
    - agentultra 3y ago
      > Many undergraduates also confidently regurgitate incorrect proofs of linear algebra theorems, do you consider them completely lacking in reasoning ability? No. Because I can ask them questions about their proof, they understand what it means, and can correct it on their own. I've seen LLM's correct their answers after receiving prompts that point out the errors in prior outputs. However I've also seen them give more wrong answers. It tells me that they don't "understand" what it means for an expression to be true or how to derive expressions. For that we'd need some form of deductive reasoning; not generating the next likely token based off a model trained on some input corpus. That's not how most mathematicians seem to do their work. However I think it seems plausible we will have a machine learning algorithm that can do simple inductive proofs and that will be nice. To the original article it seems like they're taking a first step with this. In the mean time why should anyone believe that an LLM is capable of deductive reasoning? Is a tensor enough to represent semantics to be able to dispatch a theorem to an LLM and have it write a proof? Or do I need to train it on enough proofs first before it can start inferring proof-like text?