3 ms·

I don't like your parent comment but there's merit in avoiding people believing those things actually think and are actually intelligent. In "thinking", there's

by jraph 3y ago

I don't like your parent comment but there's merit in avoiding people believing those things actually think and are actually intelligent. In "thinking", there's reasoning, which those things currently don't. They output text statistically and that's what mass media should tell their audience.

ben_w 3y ago
What is "reasoning"? Wikipedia surprised me with this opening paragraph: > Reason is the capacity of applying logic consciously by drawing conclusions from new or existing information, with the aim of seeking the truth, with more than 10 or more than 3 evidence, 99.9% reliability or 87.5% reliability. Apart from the words "consciously" (which has 100 or so different definitions, some of which do and some of which do not apply to AI), and "aim" which can be contentious depending upon if you feel that requires consciousness or can include training rewards… I think the best of them is at the lower of those two mysterious percentages I've never seen discussed before. And that's despite LLMs being wildly bad at, specifically, logic. Though even this specific weakness in general seems like a very easy thing to get around by hybridising it with the computer it has to run on e.g. by telling it how to use a compiler, as logic is the foundation of the arithmetic that the transformers use to do natural language comprehension in the first place.
- vikramkr 3y ago
  That opening for the Wikipedia article looks problematic - it's grammatically nonsense and has random numbers in it - someone should probably open a talk discussion on what happened Edit: yeah that looks like a recent edit. I thought Wikipedia was supposed to be pretty good about flagging this sort of stuff? Does anyone know how to raise the problem to those super editors that run the site? I assume they won't appreciate a random person undoing an edit https://en.wikipedia.org/w/index.php?title=Reason&diff=prev&oldid=1179925542 https://en.wikipedia.org/w/index.php?title=Reason&diff=prev&...
  - ben_w 3y ago
    If I was on desktop right now, I'd put the new bit after the existing citation and add a new {{citation needed}} for the addition. Don't worry too much about diving in, so long as you comment your changes appropriately when committing, that's the point of the place and how it does the quality control in the first place :)
    - vikramkr 3y ago
      Leaving a citation needed doesn't sound like the right thing to do for the definition of something as big as reason - esp with the grammatical confusion idk what you'd even edit it to to make sense. Like what is a three evidence or a reliability and what do those reliability numbers mean. And the sentence describing those numbers doesn't make sense - is it 10 or 3 or which of those percentages? Idk maybe it should just be undone
      - ben_w 3y ago
        Then be bold with the edits and delete it, just say why in the comment box. The deletion itself is as easy to undo as to perform, and if you say why you're doing it, someone bringing that part back can do a better job next time. :)
- Sharlin 3y ago
  > And that's despite LLMs being wildly bad at, specifically, logic. I'd say "wildly bad" is wildly exaggerated given that GPT-4 exists, even if you define "logic" as "logical puzzles encoded in natural language" and not "the thing that allows an LLM to reason highly efficiently about what token comes next".
  - ben_w 3y ago
    The numbers in Table 1 of this paper are mostly worse than I remembered, but I would also note that humans would fail the "can they reason?" standard I quoted, given the grades the species gets on several of the tests: https://arxiv.org/pdf/2304.03439.pdf https://arxiv.org/pdf/2304.03439.pdf Table 2 likewise shows significant weaknesses in GPT-4, worse than I remember, except on MED which was better than I remember it being capable of. > even if you define "logic" as "logical puzzles encoded in natural language" and not "the thing that allows an LLM to reason highly efficiently about what token comes next". I think this is a reasonable thing to say of LLMs specifically[0], given I think most people would agree with the sentence "humans are bad at quantum mechanics" even though that's what our cell chemistry is implemented on. [0] Though not AI in general! One of my recently-developed bug-bears is the conflation of the limits of LLMs with AI even in technical circles. Human brains aren't just a language cortex, there's no reason to look at an isolated language model and assume an AI can't be wired into other things, yet this assumption is baked into quite a lot of talking points even on HN.
Sharlin 3y ago
What is, then, the qualitative difference between a GOFAI automated reasoning system [1], like Coq, and an LLM? Why does Coq do reasoning but GPT-4 doesn't? Does it only count as reasoning if hand-coded by a human? Or did you have another definition of "reasoning" in mind? Statistical or probabilistic reasoning is still reasoning, and indeed essentially mandatory to make sense of the real world, and is what humans do. Indeed the failure of painstakingly hand-trained symbolic reasoning engines to model reality (as opposed to just simple toy worlds) was the entire reason for the rise of neural networks. The entire point of a neural network is to efficiently compress large amounts of statistical data into a model with as much predictive power as possible. Somewhere within GPT-4's many layers there has to exist something like Bayes networks and other ultra-efficient representations of the vast network of correlations present in the training corpus – otherwise there's no way it could do what it does. [1] https://en.wikipedia.org/wiki/Automated_reasoning https://en.wikipedia.org/wiki/Automated_reasoning
- jraph 3y ago
  I didn't know "automated reasoning" was an actual technical term. I didn't mean this by "reasoning". To me, Coq does not reason (in the meaning I intended). It checks proofs are well-formed and assists proof writing. But yes, there's some implementation of logic in Coq. It applies (a specific form of) logic. Again, LLMs output probable text. They don't do logic. They do statistics on language. Coq works on the actual objects themselves directly, not on the text that presents them. LLMs usually fail to answer to riddles correctly for this reason. We saw plenty of occurrences of this in recent times. Each time LLMs are benchmarked, people think of trying riddles and it works to some extent because something that looks like these riddles are in the training set but it eventually fails. Humans also make mistakes when solving riddles, but not for the same reasons (the process is not the same, they are not "processing text", the are really processing logic, even if the process is fallible).
  - Sharlin 3y ago
    So you just replaced the nebulously defined word "think" with an equally nebulously defined word, "reason"? Not a very convincing argument. You do not know what humans do, no human does, as we don't have such an access to our internal representations. You neither know what LLMs do, no human does, as even though we have full access to their internals, we have little to no idea how to interpret said internals. The most interesting thing about LLMs is the degree that they have learned to model the real world while only being exposed to words. Because those words are highly correlated not just with each other, but with an external entity unknown to the LLM, it's very likely that by learning to model that external entity, the LLM's word-predicting power will increase. LLMs are likely to have converged on a world-model simply because it makes the most sense given the available evidence. This is no different from how human scientists create models of things they cannot directly see based on what they can see. And of course, "seeing" is not directly perceiving the external reality either! Sensory data merely correlates with what's really going on, and the brain has learned to model the world – an unknown external entity – based on the correlations that exist in the sensory language.

by jraph 3y ago

ben_w 3y ago

What is "reasoning"? Wikipedia surprised me with this opening paragraph: > Reason is the capacity of applying logic consciously by drawing conclusions from new or existing information, with the aim of seeking the truth, with more than 10 or more than 3 evidence, 99.9% reliability or 87.5% reliability. Apart from the words "consciously" (which has 100 or so different definitions, some of which do and some of which do not apply to AI), and "aim" which can be contentious depending upon if you feel that requires consciousness or can include training rewards… I think the best of them is at the lower of those two mysterious percentages I've never seen discussed before. And that's despite LLMs being wildly bad at, specifically, logic. Though even this specific weakness in general seems like a very easy thing to get around by hybridising it with the computer it has to run on e.g. by telling it how to use a compiler, as logic is the foundation of the arithmetic that the transformers use to do natural language comprehension in the first place.

vikramkr 3y ago
That opening for the Wikipedia article looks problematic - it's grammatically nonsense and has random numbers in it - someone should probably open a talk discussion on what happened Edit: yeah that looks like a recent edit. I thought Wikipedia was supposed to be pretty good about flagging this sort of stuff? Does anyone know how to raise the problem to those super editors that run the site? I assume they won't appreciate a random person undoing an edit https://en.wikipedia.org/w/index.php?title=Reason&diff=prev&oldid=1179925542 https://en.wikipedia.org/w/index.php?title=Reason&diff=prev&...
- ben_w 3y ago
  If I was on desktop right now, I'd put the new bit after the existing citation and add a new {{citation needed}} for the addition. Don't worry too much about diving in, so long as you comment your changes appropriately when committing, that's the point of the place and how it does the quality control in the first place :)
  - vikramkr 3y ago
    Leaving a citation needed doesn't sound like the right thing to do for the definition of something as big as reason - esp with the grammatical confusion idk what you'd even edit it to to make sense. Like what is a three evidence or a reliability and what do those reliability numbers mean. And the sentence describing those numbers doesn't make sense - is it 10 or 3 or which of those percentages? Idk maybe it should just be undone
    - ben_w 3y ago
      Then be bold with the edits and delete it, just say why in the comment box. The deletion itself is as easy to undo as to perform, and if you say why you're doing it, someone bringing that part back can do a better job next time. :)
Sharlin 3y ago
> And that's despite LLMs being wildly bad at, specifically, logic. I'd say "wildly bad" is wildly exaggerated given that GPT-4 exists, even if you define "logic" as "logical puzzles encoded in natural language" and not "the thing that allows an LLM to reason highly efficiently about what token comes next".
- ben_w 3y ago
  The numbers in Table 1 of this paper are mostly worse than I remembered, but I would also note that humans would fail the "can they reason?" standard I quoted, given the grades the species gets on several of the tests: https://arxiv.org/pdf/2304.03439.pdf https://arxiv.org/pdf/2304.03439.pdf Table 2 likewise shows significant weaknesses in GPT-4, worse than I remember, except on MED which was better than I remember it being capable of. > even if you define "logic" as "logical puzzles encoded in natural language" and not "the thing that allows an LLM to reason highly efficiently about what token comes next". I think this is a reasonable thing to say of LLMs specifically[0], given I think most people would agree with the sentence "humans are bad at quantum mechanics" even though that's what our cell chemistry is implemented on. [0] Though not AI in general! One of my recently-developed bug-bears is the conflation of the limits of LLMs with AI even in technical circles. Human brains aren't just a language cortex, there's no reason to look at an isolated language model and assume an AI can't be wired into other things, yet this assumption is baked into quite a lot of talking points even on HN.

Sharlin 3y ago

What is, then, the qualitative difference between a GOFAI automated reasoning system [1], like Coq, and an LLM? Why does Coq do reasoning but GPT-4 doesn't? Does it only count as reasoning if hand-coded by a human? Or did you have another definition of "reasoning" in mind? Statistical or probabilistic reasoning is still reasoning, and indeed essentially mandatory to make sense of the real world, and is what humans do. Indeed the failure of painstakingly hand-trained symbolic reasoning engines to model reality (as opposed to just simple toy worlds) was the entire reason for the rise of neural networks. The entire point of a neural network is to efficiently compress large amounts of statistical data into a model with as much predictive power as possible. Somewhere within GPT-4's many layers there has to exist something like Bayes networks and other ultra-efficient representations of the vast network of correlations present in the training corpus – otherwise there's no way it could do what it does. [1] https://en.wikipedia.org/wiki/Automated_reasoning https://en.wikipedia.org/wiki/Automated_reasoning

jraph 3y ago
I didn't know "automated reasoning" was an actual technical term. I didn't mean this by "reasoning". To me, Coq does not reason (in the meaning I intended). It checks proofs are well-formed and assists proof writing. But yes, there's some implementation of logic in Coq. It applies (a specific form of) logic. Again, LLMs output probable text. They don't do logic. They do statistics on language. Coq works on the actual objects themselves directly, not on the text that presents them. LLMs usually fail to answer to riddles correctly for this reason. We saw plenty of occurrences of this in recent times. Each time LLMs are benchmarked, people think of trying riddles and it works to some extent because something that looks like these riddles are in the training set but it eventually fails. Humans also make mistakes when solving riddles, but not for the same reasons (the process is not the same, they are not "processing text", the are really processing logic, even if the process is fallible).
- Sharlin 3y ago
  So you just replaced the nebulously defined word "think" with an equally nebulously defined word, "reason"? Not a very convincing argument. You do not know what humans do, no human does, as we don't have such an access to our internal representations. You neither know what LLMs do, no human does, as even though we have full access to their internals, we have little to no idea how to interpret said internals. The most interesting thing about LLMs is the degree that they have learned to model the real world while only being exposed to words. Because those words are highly correlated not just with each other, but with an external entity unknown to the LLM, it's very likely that by learning to model that external entity, the LLM's word-predicting power will increase. LLMs are likely to have converged on a world-model simply because it makes the most sense given the available evidence. This is no different from how human scientists create models of things they cannot directly see based on what they can see. And of course, "seeing" is not directly perceiving the external reality either! Sensory data merely correlates with what's really going on, and the brain has learned to model the world – an unknown external entity – based on the correlations that exist in the sensory language.