5 ms·

That rephrases this: https://en.wikipedia.org/wiki/Chinese_room https://en.wikipedia.org/wiki/Chinese_room

by ae35 10y ago

samirillian 10y ago
Which rephrases this https://en.m.wikipedia.org/wiki/Leibniz's_gap https://en.m.wikipedia.org/wiki/Leibniz's_gap
Aardwolf 10y ago
This thought experiment seems to say: "the human following the instructions doesn't understand the Chinese, therefore strong AI is impossible". Seems like a fallacy to draw that conclusion, it doesn't prove anything, this is like saying: this tomato is not yellow, so bananas are not yellow. In addition, how can he say strong AI (which he defines as "understanding") isn't possible, doesn't he have at least one example of it, which is his own mind?
- jhbadger 10y ago
  Taken literally, it even suggests human intelligence is impossible. After all, our thoughts originate from a bunch of neurons following instructions and passing information between each other. And each individual neuron isn't any more intelligent than any other cell and can't be said to "understand" our thoughts any more than the man in the Chinese Room understands Chinese.
- drostie 10y ago
  Hi! Searle fan, here. Searle is not, as you might at first anticipate when you see that he's on the "computers can't be conscious" side of the argument, some sort of woo-woo psychic-spiritualism philosopher; he's actually really grounded and relatively materialistic, though he despises the latter term. (He thinks that you should not respond to the problems of dualism by choosing one or the other side of the duality to commit to, and argue the other into nonexistence -- but rather by saying "oh I guess it was stupid to draw a line here in the first place.") Even the earlier simplification I gave, he "thinks computers can't be conscious", he'd object to directly. He thinks that you're a computer, and he thinks that you're conscious. He just doesn't think that you're conscious by virtue of being the computer that you are. So that's who we're talking about here. Now, Searle says: we know, or think we know, that the brain is conscious, and it's conscious due to some aspect of the dance of molecules that's going on under the hood. And we happen to have this model of dances, called computation, which is defined in terms of abstract symbol-shuffling: the great strength of computation is that the 0s and 1s can be voltages and electric currents (as in transistors) or magnetic alignments of spins (as in hard drive) or how a gear is turned (as in the difference engine) or whether a resistance is infinite or low (as in your keyboard). They're just abstract symbols. At its most abstract the Turing test suggests that any human language can be boiled down to these abstract intrinsically-meaningless symbols (which is almost certainly true; that's the premise of the invention of writing), and then that anything which can output symbols in a way that's indistinguishable from a human being, deserves to be given the title of "understanding" that language. Searle steps in and says, no, that's crazy. And he says it's crazy because he's a philosopher and philosophers are very concerned with what words mean exactly, and the word understanding means exactly that these things you're talking about are not abstract symbols, but a computer necessarily treats them that way; it's in the definition of computation that the symbols are abstract. Now that you understand who this is and where he's coming from, maybe you can understand the argument better. He's trying to try to give a more formal proof of this above statement. So, he steps in with this Chinese room argument. The Chinese Room argument says: Your brain is conscious, because of the dance that its neurons do. Now according to this nice computer-reductionalist view, anything which does any dance is understanding a language as long as it produces results which are indistinguishable from a human's responses. So Searle says: let's step inside some Turing machine's dance so our brains are doing two dances at once. The very asset which makes computers awesome allows us to take a Turing-test-passing-algorithm and perform it ourselves: we are the computer, we get this stream of symbols in, we look up rules in some complicated rulebooks, we stream some symbols out, and we now pass the Turing test for understanding Chinese. But is the test correct: do we actually understand Chinese? No. There is no reasonable definition by which we understand Chinese just by performing this complicated computational dance. And this applies even if we move all of the rulebooks and whatever into your skull so that the neurons do those things. Now here's where we come to the conclusion that strong AI is impossible: the human has everything which is part of the definition of computation. I'll repeat that twice more. The human embodies a whole, complete, computer. There's nothing which a computer can be (by virtue of being a computer) which the human is not. So if we go this far and we say "that brain does not understand Chinese" we have to carry that over; "that silicon does not understand Chinese, either." And no computer will understand Chinese merely by virtue of running this computer program. Clearly our understanding of Chinese is not a property of arbitrary dances of neurons which can solve a Turing test, but there is something special about the right dance of neurons which solves a Turing test. The mere fact that the right dance exists points to some physics which is not reducible to computation. In other words Searle thinks that you can simulate people as much as you please, but that simulations are not reality, thank you very much. Just like we humans are very good at simulating pain-behavior without actually being in pain, so too can a computer simulate understanding-behavior without actually understanding a thing. If you want to understand how understanding works, computers will doubtless be a useful tool, but they cannot offer a complete and final answer precisely because they do not pin down the material dance firmly enough to get to something causal.
  - justinhj 10y ago
    This is the most interesting view of the Chinese Room argument I've read so far. Thank you for sharing.
  - arjunnarayan 10y ago
    Does this mean that the human simulating the Chinese is a Chinese p-zombie?
    - drostie 10y ago
      I have no clue what Searle would say, and my degree is in physics not philosophy. But my best guess is that it's probably a borderline case? The person inside the Chinese room still of course has access to qualia, and those qualia do line up in a one-to-one way with the outside world: this suggests clearly "not a p-zombie." But it does seem that in an important sense they're not the right qualia; to use that language, "there's something that it feels like to talk about a waterfall, and the person in the Chinese room does not feel that when they talk about a waterfall." If you want to up the geekery factor to eleven, you could probably think of the Chinese room as a sort of homomorphic encryption.
      - arjunnarayan 10y ago
        Thank you for both these comments, this is the best explanation I've ever read about Searle (Searle is usually a punchline in my conversations with my friends[1]). I'm not completely convinced, and I think an effective rebuttal is along the lines of "there's no such thing as a privileged qualia", but I'm not smart enough to make it properly, and have much to think about for a few nights! :) [1] For instance, most recent search in my Slack for Searle returns: "There are days when all i do is purely syntactic copy-pasting into stack overflow and copy-pasting back into my terminal, with absolutely no understanding of the underlying semantics of my actions, and have such surprisingly impressive results, that i've become more and more convinced by Searle."
  - vumgl 10y ago
    Searle was never able to explain what exactly makes a human brain conscious, other than repeating that it is based on biological processes, rather than mechanical or electrical processes (which he says cannot be conscious). He just explains a mystery (consciousness) with another mystery (why is biology superior - in his view - to mechanics as a way of "doing" consciousness)
- posterboy 10y ago
  On the other hand Google translate is (was?) a clear example that word by word translations aren't intelligent or even decidable. So he's got a point, but far from a general proof.
  - jessedhillon 10y ago
    That doesn't at all address the point of the argument. Given the following - an assumption that you can generate an algorithm to express behavior indistinguishable from a human's at a given task, and - an implementation of the algorithm at a macroscopic scale, carried out by individual humans, each executing a small part of the algorithm then, where in this system would you say that an actual understanding of the task exists? Google Translate doesn't pass the first requirement.
    - posterboy 10y ago
      That assumption is too simplistic. If the algorithmic behavior was indistinguishably human behavior, and carried out by humans, it would just be human behavior. Of course machine translation doesn't pass the requirement for human behavior. Nothing does, except humans. And if newer machine translation does pass, I'd say that's humanly accomplished, by use of tools. Nobody could learn a distinct language from nothing but a dictionary, enough to fool a native speaker. The assumption is ridiculous. The dictionary isn't conscious, either way.
      - jessedhillon 10y ago
        https://en.wikipedia.org/wiki/Chinese_room https://en.wikipedia.org/wiki/Chinese_room If you have a challenge to the Chinese Room argument, go read and respond to Searle's original essay. There's no point in responding to my TL/DR version -- offered only to correct your initial misapprehensions -- with further misapprehensions.
bbctol 10y ago
The Chinese room is a silly example, in which a highly simplified physical system that doesn't appear to have conscious thought is extended to the idea that no merely physical system has conscious thought. If the structure of the Chinese room was not a simple catalog of questions and answers, but hundreds of billions of complicated cardboard-and-marble apparatus, each processing some amount of data in a way that individually seemed meaningless, but together produced cogent speech... would you still be so sure it wasn't conscious?
- blackflame7000 10y ago
  This is a good point. The chinese room seems to stem from a more antiquated system of serial processing and the analogy doesn't quite hold when we look at the massive parallelism that occurs in the human mind that gives us consciousness. Furthermore humans can still think in the absence of understanding. In fact id argue a majority thoughts by people are conjugated without full understanding of the subject of the thought.
geofft 10y ago
The trouble I've always had with this argument is that the amount of books needed to convincingly converse in Chinese would be staggeringly huge. You essentially have an entire human personality as a lookup table. It's not just that you can ask, in Chinese, What's your favorite food, and somewhere in the pages is an answer. You can ask, Tell me about your first love, and somewhere in the pages is an entire conversation. And not only would it be an entire conversation, it would be every possible conversation, depending on how the outside person replied. If it helps visualize the problem - the outside person could say, How about a game of chess?, and the inside person cannot rely on their knowledge of chess, so the books must encode every possible chess game, convincingly enough to hold up a game. Imagine how big a lookup table for chess would be. And that's just a tiny part of what a person knows. And then despite having you accept that without question, and picture it as a small room, Searle turns around and says it would not be "at all plausible" to imagine the system as being conscious. The system is almost entirely made of book. The human in the middle is almost invisible if you look at the system. That's the implausible part - that a human could even operate this Borges library. It's so unreasonable that intuition past that point doesn't give you any useful conclusions. But, if you suspend disbelief enough to accept this galaxy-sized Chinese megalibrary, it's not implausible that "conscious" is a fine adjective for an operable printout of an entire consciousness.
- logfromblammo 10y ago
  You do not need a pre-existing, galaxy-sized library. The program operator has the ability to write into blank books and move them from one shelf in the library to another. The program could, in theory, assemble a book of knowledge on something that exists entirely outside the Room, solely from knowledge gleaned from the inputs passed in from outside. Your initial set of books might simply be a script to follow to elicit additional information and index all new information such that it can be retrieved later. You would then end up with a set of core rulebooks, a whole heap of raw conversations, and several layers of indexes into them. Searle's problem is that he decided the computer was incapable of understanding. But the computer is not the program. The library is the thing that might comprise a strong AI, not the human operator. The scary thing is that the human operator, having no understanding of the Chinese characters, does not actually know what the program is doing. It could smuggle instructions through the output window, through him, and the human operator would then be surprised when burly men drag him out of the Room one day, to replace him with a dozen monkeys who have all been trained to process the book instructions faster and more efficiently. Customers presenting inputs might then be surprised to learn that there is no longer a human inside the Room, as the conversations had become so much more interesting.

That rephrases this: https://en.wikipedia.org/wiki/Chinese_room https://en.wikipedia.org/wiki/Chinese_room

by ae35 10y ago

That rephrases this: https://en.wikipedia.org/wiki/Chinese_room https://en.wikipedia.org/wiki/Chinese_room

samirillian 10y ago

Which rephrases this https://en.m.wikipedia.org/wiki/Leibniz's_gap https://en.m.wikipedia.org/wiki/Leibniz's_gap

Aardwolf 10y ago

This thought experiment seems to say: "the human following the instructions doesn't understand the Chinese, therefore strong AI is impossible". Seems like a fallacy to draw that conclusion, it doesn't prove anything, this is like saying: this tomato is not yellow, so bananas are not yellow. In addition, how can he say strong AI (which he defines as "understanding") isn't possible, doesn't he have at least one example of it, which is his own mind?

jhbadger 10y ago
Taken literally, it even suggests human intelligence is impossible. After all, our thoughts originate from a bunch of neurons following instructions and passing information between each other. And each individual neuron isn't any more intelligent than any other cell and can't be said to "understand" our thoughts any more than the man in the Chinese Room understands Chinese.
drostie 10y ago
Hi! Searle fan, here. Searle is not, as you might at first anticipate when you see that he's on the "computers can't be conscious" side of the argument, some sort of woo-woo psychic-spiritualism philosopher; he's actually really grounded and relatively materialistic, though he despises the latter term. (He thinks that you should not respond to the problems of dualism by choosing one or the other side of the duality to commit to, and argue the other into nonexistence -- but rather by saying "oh I guess it was stupid to draw a line here in the first place.") Even the earlier simplification I gave, he "thinks computers can't be conscious", he'd object to directly. He thinks that you're a computer, and he thinks that you're conscious. He just doesn't think that you're conscious by virtue of being the computer that you are. So that's who we're talking about here. Now, Searle says: we know, or think we know, that the brain is conscious, and it's conscious due to some aspect of the dance of molecules that's going on under the hood. And we happen to have this model of dances, called computation, which is defined in terms of abstract symbol-shuffling: the great strength of computation is that the 0s and 1s can be voltages and electric currents (as in transistors) or magnetic alignments of spins (as in hard drive) or how a gear is turned (as in the difference engine) or whether a resistance is infinite or low (as in your keyboard). They're just abstract symbols. At its most abstract the Turing test suggests that any human language can be boiled down to these abstract intrinsically-meaningless symbols (which is almost certainly true; that's the premise of the invention of writing), and then that anything which can output symbols in a way that's indistinguishable from a human being, deserves to be given the title of "understanding" that language. Searle steps in and says, no, that's crazy. And he says it's crazy because he's a philosopher and philosophers are very concerned with what words mean exactly, and the word understanding means exactly that these things you're talking about are not abstract symbols, but a computer necessarily treats them that way; it's in the definition of computation that the symbols are abstract. Now that you understand who this is and where he's coming from, maybe you can understand the argument better. He's trying to try to give a more formal proof of this above statement. So, he steps in with this Chinese room argument. The Chinese Room argument says: Your brain is conscious, because of the dance that its neurons do. Now according to this nice computer-reductionalist view, anything which does any dance is understanding a language as long as it produces results which are indistinguishable from a human's responses. So Searle says: let's step inside some Turing machine's dance so our brains are doing two dances at once. The very asset which makes computers awesome allows us to take a Turing-test-passing-algorithm and perform it ourselves: we are the computer, we get this stream of symbols in, we look up rules in some complicated rulebooks, we stream some symbols out, and we now pass the Turing test for understanding Chinese. But is the test correct: do we actually understand Chinese? No. There is no reasonable definition by which we understand Chinese just by performing this complicated computational dance. And this applies even if we move all of the rulebooks and whatever into your skull so that the neurons do those things. Now here's where we come to the conclusion that strong AI is impossible: the human has everything which is part of the definition of computation. I'll repeat that twice more. The human embodies a whole, complete, computer. There's nothing which a computer can be (by virtue of being a computer) which the human is not. So if we go this far and we say "that brain does not understand Chinese" we have to carry that over; "that silicon does not understand Chinese, either." And no computer will understand Chinese merely by virtue of running this computer program. Clearly our understanding of Chinese is not a property of arbitrary dances of neurons which can solve a Turing test, but there is something special about the right dance of neurons which solves a Turing test. The mere fact that the right dance exists points to some physics which is not reducible to computation. In other words Searle thinks that you can simulate people as much as you please, but that simulations are not reality, thank you very much. Just like we humans are very good at simulating pain-behavior without actually being in pain, so too can a computer simulate understanding-behavior without actually understanding a thing. If you want to understand how understanding works, computers will doubtless be a useful tool, but they cannot offer a complete and final answer precisely because they do not pin down the material dance firmly enough to get to something causal.
- justinhj 10y ago
  This is the most interesting view of the Chinese Room argument I've read so far. Thank you for sharing.
- arjunnarayan 10y ago
  Does this mean that the human simulating the Chinese is a Chinese p-zombie?
  - drostie 10y ago
    I have no clue what Searle would say, and my degree is in physics not philosophy. But my best guess is that it's probably a borderline case? The person inside the Chinese room still of course has access to qualia, and those qualia do line up in a one-to-one way with the outside world: this suggests clearly "not a p-zombie." But it does seem that in an important sense they're not the right qualia; to use that language, "there's something that it feels like to talk about a waterfall, and the person in the Chinese room does not feel that when they talk about a waterfall." If you want to up the geekery factor to eleven, you could probably think of the Chinese room as a sort of homomorphic encryption.
    - arjunnarayan 10y ago
      Thank you for both these comments, this is the best explanation I've ever read about Searle (Searle is usually a punchline in my conversations with my friends[1]). I'm not completely convinced, and I think an effective rebuttal is along the lines of "there's no such thing as a privileged qualia", but I'm not smart enough to make it properly, and have much to think about for a few nights! :) [1] For instance, most recent search in my Slack for Searle returns: "There are days when all i do is purely syntactic copy-pasting into stack overflow and copy-pasting back into my terminal, with absolutely no understanding of the underlying semantics of my actions, and have such surprisingly impressive results, that i've become more and more convinced by Searle."
- vumgl 10y ago
  Searle was never able to explain what exactly makes a human brain conscious, other than repeating that it is based on biological processes, rather than mechanical or electrical processes (which he says cannot be conscious). He just explains a mystery (consciousness) with another mystery (why is biology superior - in his view - to mechanics as a way of "doing" consciousness)
posterboy 10y ago
On the other hand Google translate is (was?) a clear example that word by word translations aren't intelligent or even decidable. So he's got a point, but far from a general proof.
- jessedhillon 10y ago
  That doesn't at all address the point of the argument. Given the following - an assumption that you can generate an algorithm to express behavior indistinguishable from a human's at a given task, and - an implementation of the algorithm at a macroscopic scale, carried out by individual humans, each executing a small part of the algorithm then, where in this system would you say that an actual understanding of the task exists? Google Translate doesn't pass the first requirement.
  - posterboy 10y ago
    That assumption is too simplistic. If the algorithmic behavior was indistinguishably human behavior, and carried out by humans, it would just be human behavior. Of course machine translation doesn't pass the requirement for human behavior. Nothing does, except humans. And if newer machine translation does pass, I'd say that's humanly accomplished, by use of tools. Nobody could learn a distinct language from nothing but a dictionary, enough to fool a native speaker. The assumption is ridiculous. The dictionary isn't conscious, either way.
    - jessedhillon 10y ago
      https://en.wikipedia.org/wiki/Chinese_room https://en.wikipedia.org/wiki/Chinese_room If you have a challenge to the Chinese Room argument, go read and respond to Searle's original essay. There's no point in responding to my TL/DR version -- offered only to correct your initial misapprehensions -- with further misapprehensions.

bbctol 10y ago

The Chinese room is a silly example, in which a highly simplified physical system that doesn't appear to have conscious thought is extended to the idea that no merely physical system has conscious thought. If the structure of the Chinese room was not a simple catalog of questions and answers, but hundreds of billions of complicated cardboard-and-marble apparatus, each processing some amount of data in a way that individually seemed meaningless, but together produced cogent speech... would you still be so sure it wasn't conscious?

blackflame7000 10y ago
This is a good point. The chinese room seems to stem from a more antiquated system of serial processing and the analogy doesn't quite hold when we look at the massive parallelism that occurs in the human mind that gives us consciousness. Furthermore humans can still think in the absence of understanding. In fact id argue a majority thoughts by people are conjugated without full understanding of the subject of the thought.

geofft 10y ago

The trouble I've always had with this argument is that the amount of books needed to convincingly converse in Chinese would be staggeringly huge. You essentially have an entire human personality as a lookup table. It's not just that you can ask, in Chinese, What's your favorite food, and somewhere in the pages is an answer. You can ask, Tell me about your first love, and somewhere in the pages is an entire conversation. And not only would it be an entire conversation, it would be every possible conversation, depending on how the outside person replied. If it helps visualize the problem - the outside person could say, How about a game of chess?, and the inside person cannot rely on their knowledge of chess, so the books must encode every possible chess game, convincingly enough to hold up a game. Imagine how big a lookup table for chess would be. And that's just a tiny part of what a person knows. And then despite having you accept that without question, and picture it as a small room, Searle turns around and says it would not be "at all plausible" to imagine the system as being conscious. The system is almost entirely made of book. The human in the middle is almost invisible if you look at the system. That's the implausible part - that a human could even operate this Borges library. It's so unreasonable that intuition past that point doesn't give you any useful conclusions. But, if you suspend disbelief enough to accept this galaxy-sized Chinese megalibrary, it's not implausible that "conscious" is a fine adjective for an operable printout of an entire consciousness.

logfromblammo 10y ago
You do not need a pre-existing, galaxy-sized library. The program operator has the ability to write into blank books and move them from one shelf in the library to another. The program could, in theory, assemble a book of knowledge on something that exists entirely outside the Room, solely from knowledge gleaned from the inputs passed in from outside. Your initial set of books might simply be a script to follow to elicit additional information and index all new information such that it can be retrieved later. You would then end up with a set of core rulebooks, a whole heap of raw conversations, and several layers of indexes into them. Searle's problem is that he decided the computer was incapable of understanding. But the computer is not the program. The library is the thing that might comprise a strong AI, not the human operator. The scary thing is that the human operator, having no understanding of the Chinese characters, does not actually know what the program is doing. It could smuggle instructions through the output window, through him, and the human operator would then be surprised when burly men drag him out of the Room one day, to replace him with a dozen monkeys who have all been trained to process the book instructions faster and more efficiently. Customers presenting inputs might then be surprised to learn that there is no longer a human inside the Room, as the conversations had become so much more interesting.