6 ms·

If you understand transformers, you’d know that they’re doing precisely that. They’re taking a sequence of tokens (symbols), manipulating them (matrix multipli

by twayt 3y ago

If you understand transformers, you’d know that they’re doing precisely that. They’re taking a sequence of tokens (symbols), manipulating them (matrix multiplication is ultimately just moving things around and re-weighting - the same operations that you call symbol manipulations can be encoded or at least approximated there) and output a sequence of other tokens (symbols) that make sense to humans. You use the term “ascertain truth” lightly. Unless you’re operating in an axiomatic system or otherwise have access to equipment to query the real world, you can’t really “ascertain truth”. Try using ChatGPT with gpt4 enabled and present it with a novel scenario with well defined rules. That scenario surely isn’t present in its training data but it will able to show signs of making inferences and breaking the problem down. It isn’t just regurgitating memorizing text.

agentultra 3y ago
Oh cool, so we can ask it to give us a proof of the Erdős–Gyárfás conjecture? I’ve seen it confidently regurgitate incorrect proofs of linear algebra theorems. I’m just not confident it’s doing the kind of reasoning needed for us to trust that it can prove theorems formally.
- Sharlin 3y ago
  Do you also deem humans incapable of reasoning unless they can prove the Erdős–Gyárfás conjecture? Like, talk about moving the goalposts!
- twayt 3y ago
  Just because it makes mistakes on a domain that may not be part of it's data and/or architectural capabilities doesn't mean it can't do what humans consider "reasoning". Once again, I implore you to come up with a working definition of "reasoning" so that we can have a real discussion about this. Many undergraduates also confidently regurgitate incorrect proofs of linear algebra theorems, do you consider them completely lacking in reasoning ability?
  - agentultra 3y ago
    > Many undergraduates also confidently regurgitate incorrect proofs of linear algebra theorems, do you consider them completely lacking in reasoning ability? No. Because I can ask them questions about their proof, they understand what it means, and can correct it on their own. I've seen LLM's correct their answers after receiving prompts that point out the errors in prior outputs. However I've also seen them give more wrong answers. It tells me that they don't "understand" what it means for an expression to be true or how to derive expressions. For that we'd need some form of deductive reasoning; not generating the next likely token based off a model trained on some input corpus. That's not how most mathematicians seem to do their work. However I think it seems plausible we will have a machine learning algorithm that can do simple inductive proofs and that will be nice. To the original article it seems like they're taking a first step with this. In the mean time why should anyone believe that an LLM is capable of deductive reasoning? Is a tensor enough to represent semantics to be able to dispatch a theorem to an LLM and have it write a proof? Or do I need to train it on enough proofs first before it can start inferring proof-like text?
    - twayt 3y ago
      I suspect you have adopted the speech patterns of people you respect criticizing LLMs of lacking “reasoning” and “understanding” capabilities without thinking about it carefully yourself. 1. How would you define these concepts so that incontrovertible evidence is even possible. Is “reasoning” or “understanding” even possible to measure? Or are we just inferring by proxy of certain signals that an underlying understanding exists? 2. Is it an existence proof? I.e we have shown one domain where it can reason, therefore reasoning is possible. Or do we have to show that it can reason on all domains that humans can reason in? 3. If you posit that it’s a qualitative evaluation akin to the Turing test, specify something concrete here and we can talk once that’s solved too.

by twayt 3y ago

agentultra 3y ago

Oh cool, so we can ask it to give us a proof of the Erdős–Gyárfás conjecture? I’ve seen it confidently regurgitate incorrect proofs of linear algebra theorems. I’m just not confident it’s doing the kind of reasoning needed for us to trust that it can prove theorems formally.

Sharlin 3y ago
Do you also deem humans incapable of reasoning unless they can prove the Erdős–Gyárfás conjecture? Like, talk about moving the goalposts!
twayt 3y ago
Just because it makes mistakes on a domain that may not be part of it's data and/or architectural capabilities doesn't mean it can't do what humans consider "reasoning". Once again, I implore you to come up with a working definition of "reasoning" so that we can have a real discussion about this. Many undergraduates also confidently regurgitate incorrect proofs of linear algebra theorems, do you consider them completely lacking in reasoning ability?
- agentultra 3y ago
  > Many undergraduates also confidently regurgitate incorrect proofs of linear algebra theorems, do you consider them completely lacking in reasoning ability? No. Because I can ask them questions about their proof, they understand what it means, and can correct it on their own. I've seen LLM's correct their answers after receiving prompts that point out the errors in prior outputs. However I've also seen them give more wrong answers. It tells me that they don't "understand" what it means for an expression to be true or how to derive expressions. For that we'd need some form of deductive reasoning; not generating the next likely token based off a model trained on some input corpus. That's not how most mathematicians seem to do their work. However I think it seems plausible we will have a machine learning algorithm that can do simple inductive proofs and that will be nice. To the original article it seems like they're taking a first step with this. In the mean time why should anyone believe that an LLM is capable of deductive reasoning? Is a tensor enough to represent semantics to be able to dispatch a theorem to an LLM and have it write a proof? Or do I need to train it on enough proofs first before it can start inferring proof-like text?
  - twayt 3y ago
    I suspect you have adopted the speech patterns of people you respect criticizing LLMs of lacking “reasoning” and “understanding” capabilities without thinking about it carefully yourself. 1. How would you define these concepts so that incontrovertible evidence is even possible. Is “reasoning” or “understanding” even possible to measure? Or are we just inferring by proxy of certain signals that an underlying understanding exists? 2. Is it an existence proof? I.e we have shown one domain where it can reason, therefore reasoning is possible. Or do we have to show that it can reason on all domains that humans can reason in? 3. If you posit that it’s a qualitative evaluation akin to the Turing test, specify something concrete here and we can talk once that’s solved too.