5 ms·

A ML system reflects back the input it is given. If that input is human language, a very good ML system will produce output that looks like human language. It

A ML system reflects back the input it is given. If that input is human language, a very good ML system will produce output that looks like human language. It is conceptually like a mirror. A good mirror, if you put a human in front of it, will produce output that looks exactly like a human. But we know it is not a human. Not because of some flaw in its representation, but because we understand how mirrors work.

roenxi 5y ago
But humans are also conceptually mirrors. For example, none of the words you are using are ones you thought up; they mostly date back to pre-Shakespearian times. Nor is the grammar you're using, which you've copied off other people. And the arguments you are using have concepts like intelligence or the mathematics of neural nets which aren't things that you came up with either; they are being mirrored off others. And yet we'd all agree you are thinking, despite just being a mirror with very little room to add anything new. So if you want to argue that a neural net is a mirror, that does not in any way disqualify it from considering something. The part of human reasoning that isn't a mirror of other ideas is small to the point where it might not actually exist.
- verygoodname 5y ago
  Humans may even "be conceptually mirrors" (whatever that may mean): this still doesn't make mirrors be humans (conceptually or otherwise). Current language models are much more closer to "Dissociated Press" (i.e. an equation that tells you which which words are more likely, given the context) than to "human thought", and the fact that humans learn by copying does not really change this.
  - vokep 5y ago
    But it does mean mirrors might be humans/human-like, if humans are actually reducable to (self-aware) mirrors.
    - verygoodname 5y ago
      Would you say that a "Markov chain"-type (e.g. Dissociated Press) language model is "self-aware" in any way? If yes, then... how, exactly? It is basically an N x N matrix of values. Current language models (GPT et al.), are qualitatively nothing fancier than this: probabilistic models which encode regularity in language and from which you can sample to get "plausible content". If a bunch of values in a matrix is "self-aware", then I guess GPT can be seen as "self-aware"; if not, then it can't. My problem is trying to imagine an N x N matrix being self-aware (like... what does that even mean in this context?). Is GPT human-like? Sure... if you stretch the meaning of "human-like" enough (it produces content that is similar to content produced by a human). Is a human GPT-like? That's harder to argue (and I don't see how your argument would support it).
      - magusdei 5y ago
        If you know what a Markov chain is then you must also know that modern language models are nothing like Markov chains. Just as an example, a Markov chain can't do causal reasoning or correctly solve unseen programming puzzles, the way GPT-3 can. As for self-awareness, your brain is an N x N -matrix in the same sense as an ANN, so surely it must be possible for one to be self-aware? Not claiming that GPT-3 is, of course.
        verygoodname 5y ago
        > If you know what a Markov chain is then you must also know that modern language models are nothing like Markov chains. Just as an example, a Markov chain can't do causal reasoning or correctly solve unseen programming puzzles, the way GPT-3 can. First, GPT-3 can hardly do "causal reasoning" beyond exploiting regularities in the content it has been trained on. That's why you get "interesting causal reasonings" such as "the word 'muslim' co-occurs a lot with 'terrorist', so these things must be causally related". Second, just because a more complicated version of a Markov chain (i.e. a probabilistic language model) can do things that a first-order Markov chain cannot does not mean that it is qualitatively different: both things are nothing more than simple mathematical models (linear algebra + sprinkles). A polynomial model can do things than a linear model cannot, but it is no closer to consciousness and self-awareness, as far as I can tell. My point is... a mathematical model is just that (a model) and, as such, cannot be "self-aware". Humans, as cybernetic agents embedded in some environment (from which they get sensory input and with which they can interact), have agency and, as such, can display the property of "self-awareness". Models, by themselves, cannot. Humans may contain systems within them that resemble a GPT-3 language model (or a Markov chain model), and such "language models" may even be required for self-awareness (it's not obvious, but let's assume it's true). *Still*, it is not the language model itself that is self-aware, but the agent that is using the model. A GPT-3 model, by itself, cannot be self-aware, because it not much more than a applying some linear algebra operations on numbers (just like e.g. a Markov chain language model). An agent containing (among many other things) something that could be approximated by a GPT-3 model within, may. > As for self-awareness, your brain is an N x N -matrix in the same sense as an ANN, so surely it must be possible for one to be self-aware? Not claiming that GPT-3 is, of course. No, it is not. My brain is an analog computation device and not a bunch of numbers. Perhaps you can approximate some aspects of how it works using numbers, probably using digital computation devices, but that is not what anyone's brain is (neither at the "hardware" level, nor at the "software" level). Also, notice that a brain always exists within a biological agent, and it is the biological agent that is (or may be) self-aware, and not "the brain".
- throwaway34241 5y ago
  For GPT-3, I think an example makes it clearer. When GPT-3 was asked the question [1]: > Explain the pun in the following joke: “What does Adult Swim call their physical retail stores? Brick and Morty.” It answered: > … The pun “Brick and Morty” alludes to the cable television network “Adult Swim”, which broadcasts a cartoon series called “Rick and Morty”, a humorous parody of “Back to the Future” and other science fiction films. “Brick and Morty” refers not to the characters in the cartoon, but to physical stores that sell DVDs and merchandise based on the cartoon. The pun conflates two meanings of “Brick and Mortar”, a brick-and-mortar store and a brick which is part of a building. It seems on track until you get to the last sentence, which demonstrates that it does not understand the pun at all. There's other examples on that site where it falls down, like adding two 5-digit numbers. Since it's able to answer well things that match the human-written text that it ingested, but falls down on simple things otherwise, that suggests that a lot what appears to be verbal understanding is actually from the (extremely large) amount of human-written source text it ingested, and not added by GPT-3 itself. [1] https://www.gwern.net/GPT-3#pun-explanations https://www.gwern.net/GPT-3#pun-explanations
  - gwern 5y ago
    Is that a good example, though? The reason I was investigating pun explanations there was precisely because that was something I expected GPT-3 to not be able to do because of how its data is preprocessed to strip away phonetics. (If you've read that far in my page, you must've also read all my material ranting about BPEs.) The puns & rhyming are striking in that they are exceptions that prove the rule about GPT-3's general linguistic capabilities, and are clearcut examples of how BPEs sabotage GPT-3: you can see how its performance falls off a cliff as soon as you hit a BPE-related task (like rhyming or alliteration or anagrams or dad jokes). It's one of the first things I look for these days whenever someone says GPT-3 doesn't do something it seems like it ought to be able to do: "is this phonetics-related, or could in any way be damaged by BPE encoding?" (And despite my warnings, people still get tripped up by it sometimes, or don't believe me - if I had a buck for everyone in the OA Slack or forums who'd tried to make GPT-3 rhyme to prove me wrong...) But I don't believe there is anything intrinsic to Transformers or self-supervised language modeling which requires this. BPEs are just a cheap performance hack that everyone does without thinking too much about. I expect that switching to character-level (or at least phonetics-aware) encodings and spending some more compute/data would fix all of the issues I diagnose.
- snowwrestler 5y ago
  Humans are net positive; that’s where all those words and grammar and ideas came from. As a practical example, Google says it sees about 15% new queries every year (queries that have never before been entered into Google). Is that a good proxy for how net positive people are in general? I don’t know, and the exact number may not be stable or even really matter. Human intelligence is kind of a red herring in this discussion, because our intelligence rides on something deeper, something we share with all animal life. Call it “will,” or “drive,” or even “soul,” the name doesn’t really matter. What matters is that we make decisions with that deeper thing, and then we apply our intelligence to execute and interpret those decisions. What’s your favorite food? Who are you attracted to? What sounds like a fun thing to do today? Why do you sometimes feel like not going to work? We can think about these answers, and how to get what we want, but we can’t really logically reason our way into changing what we want. Not all of them, anyway. That’s what is missing from ML systems, and why they are clearly tools. If you train up an ML system and then leave it alone, it just sits there until you prompt it. If you raise a human, they do NOT just sit there until you ask them to do something. And there’s a good chance they might not do what you want even if you ask, because they have “wants” as well. To go back to my mirror analogy, people don’t suddenly step out of mirrors and disagree with us. And ML systems don’t suddenly want to write a novel instead of respond to whatever question we input. They’re not alive, they’re just really good at mirroring back written language when we give them written language as an input. Edit to clarify: I’m not claiming that “will” has to be metaphysical or supernatural. Even if you believe humans are purely physical systems (as I do), we can characterize their behavior, compare it to ML systems, and see what’s missing. And more to the point, we know it is missing from ML systems since we designed those systems and we did not put it in there. We should not ignore our existing knowledge of how an ML system works when thinking about its output. Just like we shouldn’t ignore our knowledge of how mirrors work when looking at a perfect reflection of a person.

roenxi 5y ago

But humans are also conceptually mirrors. For example, none of the words you are using are ones you thought up; they mostly date back to pre-Shakespearian times. Nor is the grammar you're using, which you've copied off other people. And the arguments you are using have concepts like intelligence or the mathematics of neural nets which aren't things that you came up with either; they are being mirrored off others. And yet we'd all agree you are thinking, despite just being a mirror with very little room to add anything new. So if you want to argue that a neural net is a mirror, that does not in any way disqualify it from considering something. The part of human reasoning that isn't a mirror of other ideas is small to the point where it might not actually exist.

verygoodname 5y ago
Humans may even "be conceptually mirrors" (whatever that may mean): this still doesn't make mirrors be humans (conceptually or otherwise). Current language models are much more closer to "Dissociated Press" (i.e. an equation that tells you which which words are more likely, given the context) than to "human thought", and the fact that humans learn by copying does not really change this.
- vokep 5y ago
  But it does mean mirrors might be humans/human-like, if humans are actually reducable to (self-aware) mirrors.
  - verygoodname 5y ago
    Would you say that a "Markov chain"-type (e.g. Dissociated Press) language model is "self-aware" in any way? If yes, then... how, exactly? It is basically an N x N matrix of values. Current language models (GPT et al.), are qualitatively nothing fancier than this: probabilistic models which encode regularity in language and from which you can sample to get "plausible content". If a bunch of values in a matrix is "self-aware", then I guess GPT can be seen as "self-aware"; if not, then it can't. My problem is trying to imagine an N x N matrix being self-aware (like... what does that even mean in this context?). Is GPT human-like? Sure... if you stretch the meaning of "human-like" enough (it produces content that is similar to content produced by a human). Is a human GPT-like? That's harder to argue (and I don't see how your argument would support it).
    - magusdei 5y ago
      If you know what a Markov chain is then you must also know that modern language models are nothing like Markov chains. Just as an example, a Markov chain can't do causal reasoning or correctly solve unseen programming puzzles, the way GPT-3 can. As for self-awareness, your brain is an N x N -matrix in the same sense as an ANN, so surely it must be possible for one to be self-aware? Not claiming that GPT-3 is, of course.
      - verygoodname 5y ago
        > If you know what a Markov chain is then you must also know that modern language models are nothing like Markov chains. Just as an example, a Markov chain can't do causal reasoning or correctly solve unseen programming puzzles, the way GPT-3 can. First, GPT-3 can hardly do "causal reasoning" beyond exploiting regularities in the content it has been trained on. That's why you get "interesting causal reasonings" such as "the word 'muslim' co-occurs a lot with 'terrorist', so these things must be causally related". Second, just because a more complicated version of a Markov chain (i.e. a probabilistic language model) can do things that a first-order Markov chain cannot does not mean that it is qualitatively different: both things are nothing more than simple mathematical models (linear algebra + sprinkles). A polynomial model can do things than a linear model cannot, but it is no closer to consciousness and self-awareness, as far as I can tell. My point is... a mathematical model is just that (a model) and, as such, cannot be "self-aware". Humans, as cybernetic agents embedded in some environment (from which they get sensory input and with which they can interact), have agency and, as such, can display the property of "self-awareness". Models, by themselves, cannot. Humans may contain systems within them that resemble a GPT-3 language model (or a Markov chain model), and such "language models" may even be required for self-awareness (it's not obvious, but let's assume it's true). *Still*, it is not the language model itself that is self-aware, but the agent that is using the model. A GPT-3 model, by itself, cannot be self-aware, because it not much more than a applying some linear algebra operations on numbers (just like e.g. a Markov chain language model). An agent containing (among many other things) something that could be approximated by a GPT-3 model within, may. > As for self-awareness, your brain is an N x N -matrix in the same sense as an ANN, so surely it must be possible for one to be self-aware? Not claiming that GPT-3 is, of course. No, it is not. My brain is an analog computation device and not a bunch of numbers. Perhaps you can approximate some aspects of how it works using numbers, probably using digital computation devices, but that is not what anyone's brain is (neither at the "hardware" level, nor at the "software" level). Also, notice that a brain always exists within a biological agent, and it is the biological agent that is (or may be) self-aware, and not "the brain".
throwaway34241 5y ago
For GPT-3, I think an example makes it clearer. When GPT-3 was asked the question [1]: > Explain the pun in the following joke: “What does Adult Swim call their physical retail stores? Brick and Morty.” It answered: > … The pun “Brick and Morty” alludes to the cable television network “Adult Swim”, which broadcasts a cartoon series called “Rick and Morty”, a humorous parody of “Back to the Future” and other science fiction films. “Brick and Morty” refers not to the characters in the cartoon, but to physical stores that sell DVDs and merchandise based on the cartoon. The pun conflates two meanings of “Brick and Mortar”, a brick-and-mortar store and a brick which is part of a building. It seems on track until you get to the last sentence, which demonstrates that it does not understand the pun at all. There's other examples on that site where it falls down, like adding two 5-digit numbers. Since it's able to answer well things that match the human-written text that it ingested, but falls down on simple things otherwise, that suggests that a lot what appears to be verbal understanding is actually from the (extremely large) amount of human-written source text it ingested, and not added by GPT-3 itself. [1] https://www.gwern.net/GPT-3#pun-explanations https://www.gwern.net/GPT-3#pun-explanations
- gwern 5y ago
  Is that a good example, though? The reason I was investigating pun explanations there was precisely because that was something I expected GPT-3 to not be able to do because of how its data is preprocessed to strip away phonetics. (If you've read that far in my page, you must've also read all my material ranting about BPEs.) The puns & rhyming are striking in that they are exceptions that prove the rule about GPT-3's general linguistic capabilities, and are clearcut examples of how BPEs sabotage GPT-3: you can see how its performance falls off a cliff as soon as you hit a BPE-related task (like rhyming or alliteration or anagrams or dad jokes). It's one of the first things I look for these days whenever someone says GPT-3 doesn't do something it seems like it ought to be able to do: "is this phonetics-related, or could in any way be damaged by BPE encoding?" (And despite my warnings, people still get tripped up by it sometimes, or don't believe me - if I had a buck for everyone in the OA Slack or forums who'd tried to make GPT-3 rhyme to prove me wrong...) But I don't believe there is anything intrinsic to Transformers or self-supervised language modeling which requires this. BPEs are just a cheap performance hack that everyone does without thinking too much about. I expect that switching to character-level (or at least phonetics-aware) encodings and spending some more compute/data would fix all of the issues I diagnose.
snowwrestler 5y ago
Humans are net positive; that’s where all those words and grammar and ideas came from. As a practical example, Google says it sees about 15% new queries every year (queries that have never before been entered into Google). Is that a good proxy for how net positive people are in general? I don’t know, and the exact number may not be stable or even really matter. Human intelligence is kind of a red herring in this discussion, because our intelligence rides on something deeper, something we share with all animal life. Call it “will,” or “drive,” or even “soul,” the name doesn’t really matter. What matters is that we make decisions with that deeper thing, and then we apply our intelligence to execute and interpret those decisions. What’s your favorite food? Who are you attracted to? What sounds like a fun thing to do today? Why do you sometimes feel like not going to work? We can think about these answers, and how to get what we want, but we can’t really logically reason our way into changing what we want. Not all of them, anyway. That’s what is missing from ML systems, and why they are clearly tools. If you train up an ML system and then leave it alone, it just sits there until you prompt it. If you raise a human, they do NOT just sit there until you ask them to do something. And there’s a good chance they might not do what you want even if you ask, because they have “wants” as well. To go back to my mirror analogy, people don’t suddenly step out of mirrors and disagree with us. And ML systems don’t suddenly want to write a novel instead of respond to whatever question we input. They’re not alive, they’re just really good at mirroring back written language when we give them written language as an input. Edit to clarify: I’m not claiming that “will” has to be metaphysical or supernatural. Even if you believe humans are purely physical systems (as I do), we can characterize their behavior, compare it to ML systems, and see what’s missing. And more to the point, we know it is missing from ML systems since we designed those systems and we did not put it in there. We should not ignore our existing knowledge of how an ML system works when thinking about its output. Just like we shouldn’t ignore our knowledge of how mirrors work when looking at a perfect reflection of a person.