7 ms·

I'm not the first to say it, but the distinction over whether models do any "actual reasoning" or not seems moot to me. Whether or not they do reasoning, they a

by johnfn 4y ago

I'm not the first to say it, but the distinction over whether models do any "actual reasoning" or not seems moot to me. Whether or not they do reasoning, they answer questions with a decent degree of accuracy, and that degree of accuracy is only going up as we feed the models more data. Whether or not they "do actual reasoning" simply won't matter. They're already superhuman in some regards; I don't think that I could have coded up the solution to that problem in 5 seconds. :)

nighthawk454 4y ago
This is a sort of dangerous interpretation. The point of saying model's "don't do reasoning" is to help us understand their strengths and weaknesses. Currently, most models are objectively trained to be "Stochastic Parrots" (as a sibling comment brought up). They do the "gut feeling" answer. But the reasoning part is straight up not in their objectives. Nor is it in their ability, by observation. There's a line of thought that if we're impressed with what we have, if it just gets bigger maybe eventually 'reasoning' will just emerge as a side-effect. This is somewhat unclear and not really a strategy per se. It's kind of like saying Moore's Law will get us to quantum computers. It's not clear that what we want is a mere scale-up of what we have. > Whether or not they do reasoning, they answer questions with a decent degree of accuracy, and that degree of accuracy is only going up as we feed the models more data. Kind of. They don't so much "answer" questions as search for stuff. Current models are giant searchable memory banks with fuzzy interpolation. This interpolation gives some synthesis ability for producing "novel" answers but it's still basically searching existing knowledge. Not really "answering" things based on an understanding. As long as it's right the distinction may not matter. But the danger is a "gut feeling" model will _always_ produce an answer and _always_ sound confident. Because that's what it's trained to do: produce good-sounding stuff. If it happens to be correct, then great. But it's not logical or reasonable currently. And worse, you can't really tell which you're getting just by the output. > Whether or not they "do actual reasoning" simply won't matter. Sure it will. There's entire tasks they categorically can't do, or worse can't be trusted with, unless we can introduce reasoning or similar. > They're already superhuman in some regards; I don't think that I could have coded up the solution to that problem in 5 seconds. :) This is superhuman in the way that Google Search is. You couldn't search the entire internet that fast either, but you don't think Google Search "feels the true meaning of art" or anything.
- quotemstr 4y ago
  > There's a line of thought that if we're impressed with what we have, if it just gets bigger maybe eventually 'reasoning' will just emerge as a side-effect. This is somewhat unclear and not really a strategy per se. It's kind of like saying Moore's Law will get us to quantum computers. It's not clear that what we want is a mere scale-up of what we have. Reasoning ability really does seem to emerge from scale: https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emergent-Abilities-of-Language-Models-to-their-Sources-b9a57ac0fcf74f30a1ab9e3e36fa1dc1 https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tr...
- visarga 4y ago
  > There's a line of thought that if we're impressed with what we have, if it just gets bigger maybe eventually 'reasoning' will just emerge as a side effect. This is somewhat unclear and not really a strategy per se. A recent analysis revealed that training on code might be the reason GPT-3 acquired multi-step reasoning abilities. It doesn't do that without code. So it looks like reasoning is emerging as a side effect of code. (section 3, long article) https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emergent-Abilities-of-Language-Models-to-their-Sources-b9a57ac0fcf74f30a1ab9e3e36fa1dc1 https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tr...
  - nighthawk454 4y ago
    This is a very cool result, although I don't think it implies that scale will create reasoning. Current AIs are fuzzy mad-libs engines. They probabilistically fill in the blank, based on the statistics of all the example text they've seen in training. In order to predict code successfully, most certainly there's longer-range multi-hop patterns between tokens. (A variable multiple lines away, a variable whose meaning/type/value changes after intermediate lines, ...) Regular english has far less of this. So it's not terribly surprising to me that including code in training is more effective, relative to not training on it at all. The question then is say we train on all text ever written in all languages. Including books, code, reddit comments, everything. Could the current architectures and training objectives produce a reasoning AGI? Or are we missing something? Frankly I think no one really knows for sure yet. I suspect we're missing something and "fill in the blank" is not the key to the universe.
mrguyorama 4y ago
"Do the models do any actual reasoning" is the difference between your ML blackbox having a child's level of understanding of things where it just repeats what it's been trained on and just "monkey see monkey do" it's way to an output, or whether it's actually mixing previous input and predicting and modeling and producing an output. There's a bunch of famous research that shows a baby and toddlers have basic understanding of physics. If you give a crawling baby a small cliff but make a bridge out of glass, the baby will refuse to cross it, because it's limited understanding prevents it from knowing that the glass is safe to crawl on and it won't fall. In contrast older humans, even those with a fear of heights, are able to recognize that properly strong glass bridges are perfectly safe, and they won't fall through them just because they can see through them. What changes when you go from one to the next? Is it just more data fed into the feedback machine, or does the brain build entirely new circuits and pathways and systems to process this more complicated modeling of the world and info it gets? Everything about machine learning just assumes it's the first, with no actual science to support it, and further claims that neural nets with back-propagation are fully able to model that system, even though we have no idea how the brain corrects errors in it's modeling and a single neuron is WAY more powerful than a small section of a neural network. These are literally the same mistakes made all the time in the AI field. The field of AI made all these same claims of human levels of intelligence back when the hot new thing was "expert systems" where the plan was, surely if you make enough if/else statements, you can model a human level intelligence. When that proved dumb, we got an AI winter. There are serious open questions about neural networks and current ML that the community just flat out ignores and handwaves away, usually pretending that they are philosophy questions when they aren't. "Can a giant neural network exactly model what the human brain does" is not a philosophy question.
- visarga 4y ago
  It all boils down to having some sort of embodiment, or a way to verify. For code it would suffice to let the model generate and execute code, and learn from errors. Give it enough "experience" with code execution and it will learn on its own, like AlphaGo. Generate more data and retrain the models a few times.
- kqr 4y ago
didericis 4y ago
I strongly disagree. Humans have perceptual systems we can never fully understand for the same reasons no mathematical system can ever be provably consistent and complete. We cannot prove the reliability and accuracy of our perception with our perception. The only thing which suggests the reliability of our perception is our existence. The better ways of perceiving make a better map of reality that makes persistence more likely. Our ability to manipulate reality and achieve desired outcomes is what distinguishes good perception from bad perception. If data directed by human perception is fed into these systems, they have an amazing ability to condense and organize accurate/good faith but relatively unstructured knowledge that is entered into them. They are and will remain extremely useful because of that ability. But they do not have access to reality because they have not been grown from it through evolution. That means that fundamentally they have no error correcting beyond human input. As systems become increasingly unintelligible due to increasing the scale of the data, these systems are going to become more and more disconnected from reality, and less accurate. Think of how nearly every financial disaster occurs despite increasingly sophisticated economic models that build off of more and more data. As you get more and more abstraction needed to handle more and more data, you get more and more error. There is a reason biological systems tap out at a certain size, large organizations decay over time, most animals reproduce instead of live forever. Errors in large complex systems are what nature has been fighting for billions of years, and tend to compound in subtle and pernicious ways. Imagine a world in which AI systems are not fed carefully categorized human data, but are operating in an internet in which 5% is AI data. Then 15%. Then 50%. Then 75%. Then what human data there is gets influenced by AI content and humans doubting reality based categorizations because of social pressure/because AI is perceived to be better. Very soon you get self referential systems of AI data feeding AI and further and further distance from original source perception and categorization. Self referential group think is disastrous enough when only humans are involved. If you add machines which you cannot appeal to and are entirely deferential to statistical majorities, which then become even more entrenched self referential statistical majorities, you very quickly become entirely disconnected from any notion of reality.
trynewideas 4y ago
I want to be clear that I still find it impressive, in the same way I find Riffusion impressive. If anything, I'm looking at Norvig's pointing out that "the biggest issue is that there is no explanation of why the code is the way it is, no justification for how it works." The model can't and won't; it's an unreasonable expectation, and I can't tell whether Norvig is asking for it in good faith. If I assume he is, and his proposed suggestions that the model "participate in a conversation that leads to the kind of questions and answers we discussed here, thereby building trust in the program" and "generate documentation or tests that would build trust in the code" are also in good faith, then I maintain that he's still missing a fundamental limitation of these models even as he outlines its shape with great specificity. They literally and demonstrably are incapable of coherently doing what he wants; they can't be trained to engender trust, only to mimic actions that might by generating novel responses based on patterns. That would still not be reasoning through the problem to engineer a solution, it's just an extremely effective, superhuman con of novel mimicry. Which, again, is still really, really impressive, and even potentially useful, but in a different way than we might want or expect it to be, and in a dangerous way to use as a stable foundation for iteration toward AGI.
ookdatnog 4y ago
Reasoning is meaningfully distinct from what the models do now. You can ask ChatGPT to multiply two smallish natural numbers and often it will give the correct answer. Now look what happens if you ask it to multiply two very large numbers: > To multiply two large numbers, such as 3146746135164684 times 468473136468433, we can use the traditional multiplication algorithm. This algorithm involves breaking the numbers down into smaller groups of digits, multiplying each group by the corresponding group of digits in the other number, and then adding up the products to find the final result. > Here is an example of how to multiply 3146746135164684 times 468473136468433 using the traditional multiplication algorithm: > Break the numbers down into groups of digits. For example, we could group the digits of 3146746135164684 as 314, 6746, 135, 1646, 84 and group the digits of 468473136468433 as 4684, 7313, 6468, 433. > Multipply each group of digits by the corresponding group of digits in the other number. For example, we would multiply 314 times 4684 to get 1470196, 6746 times 7313 to get 49470158, 135 times 6468 to get 873980, 1646 times 433 to get 708338, and 84 times 1 to get 84. > Add up the products to find the final result. In this case, the final result would be 1470196 + 49470158 + 873980 + 708338 + 84 = 146845556. > Therefore, the result of multiplying 3146746135164684 times 468473136468433 using the traditional multiplication algorithm is 146845556. It's not just that the answer is wrong, is that it's complete nonsense. Reasoning is a style of thinking that scales. You may be more likely to get the wrong answer in a very long chain of reasoning because at every step you have a nonzero chance of making a mistake, but the mistake is identifiable and explainable. That's why teachers ask you to show your work. Even if you get the answer wrong, they can see at a glance whether you understand the material or not. We can see at a glance that ChatGPT does not understand multiplication.
- johnfn 4y ago
  I don't think I buy this argument. ChatGPT seems to understand how to reason about a large multiplication the same that a 6 or 7 year old might, and I would expect a 6 or 7 year old to make similarly large errors. No one claims that 6 or 7 year olds are unable to reason.
  - gonehome 4y ago
    Yeah, in the original gpt-3 paper one of the more interesting bits was that it made similar off by one errors a human would make when doing arithmetic (and they controlled for memorized test data).
paganel 4y ago
> and that degree of accuracy is only going up as we feed the models more data. The problems comes when the data that is fed is of the "Hitler did nothing wrong"-type. That AI system will have no problem regurgitant something that takes that at face value, while a thinking individual knows it to be false. There's also the issue of what do you do if the data being fed is only "valid" for people that happen to have a certain skin colour? Or a certain ethnicity? Or a certain gender? Or a specific socio-economic status? There's a great short story about a "robot" ingurgitating lots and lots of data with no extrinsic value in Stanislaw Lem's "The Cyberiad" (minus the Hitler part). People like Norvig are smart enough to give lots and lots of references in order to prove their point but they're not smart enough to see the bigger picture (the one pointed to by people like Lem).
- riwsky 4y ago
  "They are vulnerable to reproducing poor quality training data" (Peter Norvig, in the article)
  - paganel 4y ago
    I, of course, didn’t go trough all that huge article. Glad that Norvig is aware of it.
Barrin92 4y ago
> Whether or not they "do actual reasoning" simply won't matter. It does. This is trivially true in some domains like mathematics. If you're going to try and measure the last number of pi (which gpt-3 a while ago thought was 3 apparently because some python library returned that result) from observed data I wish you good luck. Deductive reasoning is a necessary skill for any generally intelligent agent. Now how to get from current AI to a system that can reason and how much we're going to need to put in there is an open question, but to deny that deduction is necessary is kind of trivially false. This almost harkens back to the naive empiricism of Skinner that died with the cognitive sciences.
reaperducer 4y ago
Whether or not they do reasoning, they answer questions with a decent degree of accuracy How do you know they're accurate if they can't explain how they got the answer?

by johnfn 4y ago

nighthawk454 4y ago

This is a sort of dangerous interpretation. The point of saying model's "don't do reasoning" is to help us understand their strengths and weaknesses. Currently, most models are objectively trained to be "Stochastic Parrots" (as a sibling comment brought up). They do the "gut feeling" answer. But the reasoning part is straight up not in their objectives. Nor is it in their ability, by observation. There's a line of thought that if we're impressed with what we have, if it just gets bigger maybe eventually 'reasoning' will just emerge as a side-effect. This is somewhat unclear and not really a strategy per se. It's kind of like saying Moore's Law will get us to quantum computers. It's not clear that what we want is a mere scale-up of what we have. > Whether or not they do reasoning, they answer questions with a decent degree of accuracy, and that degree of accuracy is only going up as we feed the models more data. Kind of. They don't so much "answer" questions as search for stuff. Current models are giant searchable memory banks with fuzzy interpolation. This interpolation gives some synthesis ability for producing "novel" answers but it's still basically searching existing knowledge. Not really "answering" things based on an understanding. As long as it's right the distinction may not matter. But the danger is a "gut feeling" model will _always_ produce an answer and _always_ sound confident. Because that's what it's trained to do: produce good-sounding stuff. If it happens to be correct, then great. But it's not logical or reasonable currently. And worse, you can't really tell which you're getting just by the output. > Whether or not they "do actual reasoning" simply won't matter. Sure it will. There's entire tasks they categorically can't do, or worse can't be trusted with, unless we can introduce reasoning or similar. > They're already superhuman in some regards; I don't think that I could have coded up the solution to that problem in 5 seconds. :) This is superhuman in the way that Google Search is. You couldn't search the entire internet that fast either, but you don't think Google Search "feels the true meaning of art" or anything.

quotemstr 4y ago
> There's a line of thought that if we're impressed with what we have, if it just gets bigger maybe eventually 'reasoning' will just emerge as a side-effect. This is somewhat unclear and not really a strategy per se. It's kind of like saying Moore's Law will get us to quantum computers. It's not clear that what we want is a mere scale-up of what we have. Reasoning ability really does seem to emerge from scale: https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emergent-Abilities-of-Language-Models-to-their-Sources-b9a57ac0fcf74f30a1ab9e3e36fa1dc1 https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tr...
visarga 4y ago
> There's a line of thought that if we're impressed with what we have, if it just gets bigger maybe eventually 'reasoning' will just emerge as a side effect. This is somewhat unclear and not really a strategy per se. A recent analysis revealed that training on code might be the reason GPT-3 acquired multi-step reasoning abilities. It doesn't do that without code. So it looks like reasoning is emerging as a side effect of code. (section 3, long article) https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emergent-Abilities-of-Language-Models-to-their-Sources-b9a57ac0fcf74f30a1ab9e3e36fa1dc1 https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tr...
- nighthawk454 4y ago
  This is a very cool result, although I don't think it implies that scale will create reasoning. Current AIs are fuzzy mad-libs engines. They probabilistically fill in the blank, based on the statistics of all the example text they've seen in training. In order to predict code successfully, most certainly there's longer-range multi-hop patterns between tokens. (A variable multiple lines away, a variable whose meaning/type/value changes after intermediate lines, ...) Regular english has far less of this. So it's not terribly surprising to me that including code in training is more effective, relative to not training on it at all. The question then is say we train on all text ever written in all languages. Including books, code, reddit comments, everything. Could the current architectures and training objectives produce a reasoning AGI? Or are we missing something? Frankly I think no one really knows for sure yet. I suspect we're missing something and "fill in the blank" is not the key to the universe.

mrguyorama 4y ago

"Do the models do any actual reasoning" is the difference between your ML blackbox having a child's level of understanding of things where it just repeats what it's been trained on and just "monkey see monkey do" it's way to an output, or whether it's actually mixing previous input and predicting and modeling and producing an output. There's a bunch of famous research that shows a baby and toddlers have basic understanding of physics. If you give a crawling baby a small cliff but make a bridge out of glass, the baby will refuse to cross it, because it's limited understanding prevents it from knowing that the glass is safe to crawl on and it won't fall. In contrast older humans, even those with a fear of heights, are able to recognize that properly strong glass bridges are perfectly safe, and they won't fall through them just because they can see through them. What changes when you go from one to the next? Is it just more data fed into the feedback machine, or does the brain build entirely new circuits and pathways and systems to process this more complicated modeling of the world and info it gets? Everything about machine learning just assumes it's the first, with no actual science to support it, and further claims that neural nets with back-propagation are fully able to model that system, even though we have no idea how the brain corrects errors in it's modeling and a single neuron is WAY more powerful than a small section of a neural network. These are literally the same mistakes made all the time in the AI field. The field of AI made all these same claims of human levels of intelligence back when the hot new thing was "expert systems" where the plan was, surely if you make enough if/else statements, you can model a human level intelligence. When that proved dumb, we got an AI winter. There are serious open questions about neural networks and current ML that the community just flat out ignores and handwaves away, usually pretending that they are philosophy questions when they aren't. "Can a giant neural network exactly model what the human brain does" is not a philosophy question.

visarga 4y ago
It all boils down to having some sort of embodiment, or a way to verify. For code it would suffice to let the model generate and execute code, and learn from errors. Give it enough "experience" with code execution and it will learn on its own, like AlphaGo. Generate more data and retrain the models a few times.
kqr 4y ago

didericis 4y ago

I strongly disagree. Humans have perceptual systems we can never fully understand for the same reasons no mathematical system can ever be provably consistent and complete. We cannot prove the reliability and accuracy of our perception with our perception. The only thing which suggests the reliability of our perception is our existence. The better ways of perceiving make a better map of reality that makes persistence more likely. Our ability to manipulate reality and achieve desired outcomes is what distinguishes good perception from bad perception. If data directed by human perception is fed into these systems, they have an amazing ability to condense and organize accurate/good faith but relatively unstructured knowledge that is entered into them. They are and will remain extremely useful because of that ability. But they do not have access to reality because they have not been grown from it through evolution. That means that fundamentally they have no error correcting beyond human input. As systems become increasingly unintelligible due to increasing the scale of the data, these systems are going to become more and more disconnected from reality, and less accurate. Think of how nearly every financial disaster occurs despite increasingly sophisticated economic models that build off of more and more data. As you get more and more abstraction needed to handle more and more data, you get more and more error. There is a reason biological systems tap out at a certain size, large organizations decay over time, most animals reproduce instead of live forever. Errors in large complex systems are what nature has been fighting for billions of years, and tend to compound in subtle and pernicious ways. Imagine a world in which AI systems are not fed carefully categorized human data, but are operating in an internet in which 5% is AI data. Then 15%. Then 50%. Then 75%. Then what human data there is gets influenced by AI content and humans doubting reality based categorizations because of social pressure/because AI is perceived to be better. Very soon you get self referential systems of AI data feeding AI and further and further distance from original source perception and categorization. Self referential group think is disastrous enough when only humans are involved. If you add machines which you cannot appeal to and are entirely deferential to statistical majorities, which then become even more entrenched self referential statistical majorities, you very quickly become entirely disconnected from any notion of reality.

trynewideas 4y ago

I want to be clear that I still find it impressive, in the same way I find Riffusion impressive. If anything, I'm looking at Norvig's pointing out that "the biggest issue is that there is no explanation of why the code is the way it is, no justification for how it works." The model can't and won't; it's an unreasonable expectation, and I can't tell whether Norvig is asking for it in good faith. If I assume he is, and his proposed suggestions that the model "participate in a conversation that leads to the kind of questions and answers we discussed here, thereby building trust in the program" and "generate documentation or tests that would build trust in the code" are also in good faith, then I maintain that he's still missing a fundamental limitation of these models even as he outlines its shape with great specificity. They literally and demonstrably are incapable of coherently doing what he wants; they can't be trained to engender trust, only to mimic actions that might by generating novel responses based on patterns. That would still not be reasoning through the problem to engineer a solution, it's just an extremely effective, superhuman con of novel mimicry. Which, again, is still really, really impressive, and even potentially useful, but in a different way than we might want or expect it to be, and in a dangerous way to use as a stable foundation for iteration toward AGI.

ookdatnog 4y ago

Reasoning is meaningfully distinct from what the models do now. You can ask ChatGPT to multiply two smallish natural numbers and often it will give the correct answer. Now look what happens if you ask it to multiply two very large numbers: > To multiply two large numbers, such as 3146746135164684 times 468473136468433, we can use the traditional multiplication algorithm. This algorithm involves breaking the numbers down into smaller groups of digits, multiplying each group by the corresponding group of digits in the other number, and then adding up the products to find the final result. > Here is an example of how to multiply 3146746135164684 times 468473136468433 using the traditional multiplication algorithm: > Break the numbers down into groups of digits. For example, we could group the digits of 3146746135164684 as 314, 6746, 135, 1646, 84 and group the digits of 468473136468433 as 4684, 7313, 6468, 433. > Multipply each group of digits by the corresponding group of digits in the other number. For example, we would multiply 314 times 4684 to get 1470196, 6746 times 7313 to get 49470158, 135 times 6468 to get 873980, 1646 times 433 to get 708338, and 84 times 1 to get 84. > Add up the products to find the final result. In this case, the final result would be 1470196 + 49470158 + 873980 + 708338 + 84 = 146845556. > Therefore, the result of multiplying 3146746135164684 times 468473136468433 using the traditional multiplication algorithm is 146845556. It's not just that the answer is wrong, is that it's complete nonsense. Reasoning is a style of thinking that scales. You may be more likely to get the wrong answer in a very long chain of reasoning because at every step you have a nonzero chance of making a mistake, but the mistake is identifiable and explainable. That's why teachers ask you to show your work. Even if you get the answer wrong, they can see at a glance whether you understand the material or not. We can see at a glance that ChatGPT does not understand multiplication.

johnfn 4y ago
I don't think I buy this argument. ChatGPT seems to understand how to reason about a large multiplication the same that a 6 or 7 year old might, and I would expect a 6 or 7 year old to make similarly large errors. No one claims that 6 or 7 year olds are unable to reason.
- gonehome 4y ago
  Yeah, in the original gpt-3 paper one of the more interesting bits was that it made similar off by one errors a human would make when doing arithmetic (and they controlled for memorized test data).

paganel 4y ago

> and that degree of accuracy is only going up as we feed the models more data. The problems comes when the data that is fed is of the "Hitler did nothing wrong"-type. That AI system will have no problem regurgitant something that takes that at face value, while a thinking individual knows it to be false. There's also the issue of what do you do if the data being fed is only "valid" for people that happen to have a certain skin colour? Or a certain ethnicity? Or a certain gender? Or a specific socio-economic status? There's a great short story about a "robot" ingurgitating lots and lots of data with no extrinsic value in Stanislaw Lem's "The Cyberiad" (minus the Hitler part). People like Norvig are smart enough to give lots and lots of references in order to prove their point but they're not smart enough to see the bigger picture (the one pointed to by people like Lem).

riwsky 4y ago
"They are vulnerable to reproducing poor quality training data" (Peter Norvig, in the article)
- paganel 4y ago
  I, of course, didn’t go trough all that huge article. Glad that Norvig is aware of it.

Barrin92 4y ago

> Whether or not they "do actual reasoning" simply won't matter. It does. This is trivially true in some domains like mathematics. If you're going to try and measure the last number of pi (which gpt-3 a while ago thought was 3 apparently because some python library returned that result) from observed data I wish you good luck. Deductive reasoning is a necessary skill for any generally intelligent agent. Now how to get from current AI to a system that can reason and how much we're going to need to put in there is an open question, but to deny that deduction is necessary is kind of trivially false. This almost harkens back to the naive empiricism of Skinner that died with the cognitive sciences.

reaperducer 4y ago

Whether or not they do reasoning, they answer questions with a decent degree of accuracy How do you know they're accurate if they can't explain how they got the answer?