8 ms·

Ask HN: Opinion about Searle's Chinese Room?

12 points by wsieroci 11y ago

Hi, I would like to know your opinion about John Searle's Chinese Room argument against "Strong AI". Best, Wiktor

lazyant 11y ago
we had a thread just yesterday https://news.ycombinator.com/item?id=10867791 https://news.ycombinator.com/item?id=10867791 and also last month https://news.ycombinator.com/item?id=10740748 https://news.ycombinator.com/item?id=10740748
- p4wnc6 11y ago
  I'm incredibly surprised that out of both previous threads, the piece "Why Philosophers Should Care About Computational Complexity" by Scott Aaronson [1] was only mentioned in one comment, and seemed to spark no further discussion. Aaronson dedicates significant space specifically to the Chinese Room problem, and has a good literature review of different computer science objections (my favorite was probably the one estimating the entropy of English and coming up with space bounds on the size of all 5-minute long conversations in English). It is one of the most comprehensive takedowns of the Chinese Room problem. Along similar lines, Aaronson discusses later in the paper the problem "Does a rock implement every Finite-State Automaton?" popularized by Chalmers [2] and many computer science rebuttals. [1] http://www.scottaaronson.com/papers/philos.pdf http://www.scottaaronson.com/papers/philos.pdf [2] http://consc.net/papers/rock.html http://consc.net/papers/rock.html
  - makebelieve 11y ago
    Aaronson does not critique the chinese room argument at all, at least not in the paper cited.
    - samizdatum 11y ago
      He does, on page 14.
      - makebelieve 11y ago
        Um, he doesn't. He is talking about Searle's argument, but he is not refuting it. Explicitly he says: "Where others invoked complexity considerations to argue with Searle about the metaphysical question, I’m invoking them to argue with Penrose about the practical question." Aaronson is not arguing with Searle at all, he is using Searle's argument as an example of other peoples faulty thinking about complexity. Aaronson understands (which is why he doesn't elucidate Searle's argument) that Searle is talking about meaning, and Aaronson is criticizing other peoples critics of Searle that are based in computation for failing to understand complexity: "I find this response to Searle extremely interesting—since if correct, it suggests that the distinction between polynomial and exponential complexity has metaphysical significance. According to this response, an exponential-sized lookup table that passed the Turing Test would not be sentient (or conscious, intelligent, self-aware, etc.), but a polynomially-bounded program with exactly the same input/output behavior would be sentient." More to the point, Aaronson doesn't address the meaning (or as he says, the metaphysical) questions at all. He is interested in the complexity problem of producing a machine that passes the Turing Test, and how philosophers don't seem to grok that very practical problem. Searle recognizes the practical problem for what it is (lookup tables can be Turing complete) and talks about meaning and asks us to consider where the meaning of things are, and shows that meaning does not exist in the functions or the data the functions process. So that even if a machine passes the Turing Test, it fools the observer. The machine still would not be "intelligent"; it would not be conscious.
        p4wnc6 11y ago
        Can you elaborate on when you say "lookup tables can be Turing complete" as opposed to just able to pass some given Turing test? I feel like there's some conflation about what lookup table we're talking about. Also, my reading of that passage by Aaronson is very different from yours. I read it as him saying, these computer scientists have put forward a fairly serious and compelling set of arguments that the actual complexity class of the translator algorithm has philosophical importance. Meanwhile, the analytical philosophy response is just to say something hand-wavy about "meaning." I agree Aaronson's not saying anything definitive (he is extremely conservative about making definitive claims, despite how hotly involved he becomes in the few cases when he does make definitive claims). But I don't agree that he is framing it to raise questions about the validity of the CS rebuttal papers that he cites.
dbpokorny 11y ago
In theory we could rig up a quad copter with a machine gun, loudspeaker, and AI, with programming that demands that every human it encounters be interrogated with the question, "do I, a mechanical computer, have the ability to think?" and if the human gives a negative reply, it uses the machine gun on the human until the human is dead or decides to utter a positive response: "yes, this flying death machine can think". Whether or not we say "machines can think" is a political question, and political power comes out of the barrel of a gun. If a machine can wield political power, then it can get you to say "machines can think" because truth is inherently political.
- cousin_it 11y ago
  You can also be forced to say that 2+2=5, but that doesn't make it true. "Can machines think?" demands a substantive answer.
  - dbpokorny 11y ago
    The answer to that question depends on whether or not a robot is pointing a gun at your head. Might makes right.
    - AnimalMuppet 11y ago
      Might can make people lie. That does not make the lies into the truth.
      - dbpokorny 11y ago
        I'm not qualified to continue this discussion further. https://www.youtube.com/watch?v=GicWZl9HmU4 https://www.youtube.com/watch?v=GicWZl9HmU4 https://en.wikipedia.org/wiki/Two_truths_doctrine https://en.wikipedia.org/wiki/Two_truths_doctrine
makebelieve 11y ago
Searle is making an argument about awareness. That a computer system is explicitly unaware of any of it's content. That it's programs are functions and the data also performs a purely functional role. In essence, computers cannot engage in acts of meaning. The programmers and users are engaged in acts of meaning. For instance, saying a program "has a bug", is a completely misleading statement. No programs have bugs. It is impossible for a program to have a bug, just as it is impossible for a physical process to "do something wrong". Programs do what they do, just as molecular processes do what they do. The concept of error and meaning does not exist in a program, just as it does not exist in the physical universe. Meaning (and errors and bugs are a kind of meaning) are things outside programs and outside physics. When a program "has a bug" it means the programmer screwed up, not the program. A program cannot produce errors, because programs, and computer systems in general, do not have the capacity to have meaning. This is what Searle is demonstrating with his argument. This is true for all the popular computational approaches we have today. However, because the human brain appears to function in a purely physical way, and computers function in a purely physical way, it should be theoretically possible to create a computer system that is conscious and aware of meaning just as we are. You refer to this as "Strong AI". Other refer to it as Artificial General Intelligence. I refer to this as machine consciousness. To solve the machine consciousness problem means understanding how awareness, meaning, and representation in general, works. Then building a computer system that engages in representation and instantiates awareness. If an actual person were put into Searle's box, the person would learn chinese. Also, the person could 'intentionally' produce incorrect answers annoying the "programmers" who set the box up in the first place. But a modern computer system cannot 'intentionally' produce errors. it's completely non-sensical to talk about computers as having intention at all. programmers have intention, not computers. Solving the intentionality problem is the other leg of machine consciousness. Elon Musk, Steven Hawking, Nick Bostrom and others make arguments about the dangers of an AI (of any variety) which may acquire intentionality and representational ability, while ignoring the actual deep problems embedded in acquiring those abilities. Awareness, representation, and intention are so fundamental to experience that we have a very difficult time understanding when they happen and when they do not. We see a representational world all around us, but very explicitly, there are no representations at all in the physical world. I believe machine consciousness is possible, but none of the existing approaches will get us there. Searle's chinese room is one succinct argument as to why. The approach I am taking is a kind of metabolic computing. Where single function processes interact in some way similar to molecular interactions and those processes are developed to produce, computational structures like membranes and DNA and eventually "cells". These cells then form multi-cellular structures. These multi-cellular structures and underlying "molecular" interactions instantiate representations and representational processes, like a nervous system. A computational nervous system which embodies representation, intention, sensation, action, imagination, and because it engages in representation making, would be aware. I would love to hear someone describe how any kind of computational approach can produce meaning inside a computer system. We produce meaning and representations so easily; it's hard to understand the difference of perspective necessary to see how representations must form. If someone has an easier approach than the one I am taking, I would be very interested in seeing how they solve the problems of meaning and intention with code.
cousin_it 11y ago
Some people are uncomfortable with the idea that brains can work on "mere" physics. But what's the alternative? A decree from Cthulhu saying human brains are the only devices in the universe that can communicate with consciousness, and all other objects are doomed to be dumb matter? I think that's really unlikely. So it seems like we must bite the bullet and admit that the computation implemented in the Chinese room exhibits genuine understanding. I'm not completely certain, because we don't really understand consciousness yet and we could always find new evidence, but the evidence so far points in one direction.
fallingfrog 11y ago
Well, a Chinese room can simulate a computer, a computer can run a physics simulation and a brain is a physical thing and brains are conscious. So it's pretty open and shut imho; yes, a Chinese room can be conscious. That much is clear. Now what it means to say that something is conscious is another question. I've never seen the word rigorously defined.
chrisdevereux 11y ago
Its popularity in undergraduate philosophy courses reflects the fact that it is easy to criticise more than it telling us something interesting about consciousness.
makebelieve 11y ago
you might want to watch this lecture by Searle at Google: https://www.youtube.com/watch?v=rHKwIYsPXLg https://www.youtube.com/watch?v=rHKwIYsPXLg
- wsieroci 11y ago
  I saw it and I am constantly pretty amazed just how people can't grasp his argument.

12 points by wsieroci 11y ago

Hi, I would like to know your opinion about John Searle's Chinese Room argument against "Strong AI". Best, Wiktor

we had a thread just yesterday https://news.ycombinator.com/item?id=10867791 https://news.ycombinator.com/item?id=10867791 and also last month https://news.ycombinator.com/item?id=10740748 https://news.ycombinator.com/item?id=10740748

p4wnc6 11y ago
I'm incredibly surprised that out of both previous threads, the piece "Why Philosophers Should Care About Computational Complexity" by Scott Aaronson [1] was only mentioned in one comment, and seemed to spark no further discussion. Aaronson dedicates significant space specifically to the Chinese Room problem, and has a good literature review of different computer science objections (my favorite was probably the one estimating the entropy of English and coming up with space bounds on the size of all 5-minute long conversations in English). It is one of the most comprehensive takedowns of the Chinese Room problem. Along similar lines, Aaronson discusses later in the paper the problem "Does a rock implement every Finite-State Automaton?" popularized by Chalmers [2] and many computer science rebuttals. [1] http://www.scottaaronson.com/papers/philos.pdf http://www.scottaaronson.com/papers/philos.pdf [2] http://consc.net/papers/rock.html http://consc.net/papers/rock.html
- makebelieve 11y ago
  Aaronson does not critique the chinese room argument at all, at least not in the paper cited.
  - samizdatum 11y ago
    He does, on page 14.
    - makebelieve 11y ago
      Um, he doesn't. He is talking about Searle's argument, but he is not refuting it. Explicitly he says: "Where others invoked complexity considerations to argue with Searle about the metaphysical question, I’m invoking them to argue with Penrose about the practical question." Aaronson is not arguing with Searle at all, he is using Searle's argument as an example of other peoples faulty thinking about complexity. Aaronson understands (which is why he doesn't elucidate Searle's argument) that Searle is talking about meaning, and Aaronson is criticizing other peoples critics of Searle that are based in computation for failing to understand complexity: "I find this response to Searle extremely interesting—since if correct, it suggests that the distinction between polynomial and exponential complexity has metaphysical significance. According to this response, an exponential-sized lookup table that passed the Turing Test would not be sentient (or conscious, intelligent, self-aware, etc.), but a polynomially-bounded program with exactly the same input/output behavior would be sentient." More to the point, Aaronson doesn't address the meaning (or as he says, the metaphysical) questions at all. He is interested in the complexity problem of producing a machine that passes the Turing Test, and how philosophers don't seem to grok that very practical problem. Searle recognizes the practical problem for what it is (lookup tables can be Turing complete) and talks about meaning and asks us to consider where the meaning of things are, and shows that meaning does not exist in the functions or the data the functions process. So that even if a machine passes the Turing Test, it fools the observer. The machine still would not be "intelligent"; it would not be conscious.
      - p4wnc6 11y ago
        Can you elaborate on when you say "lookup tables can be Turing complete" as opposed to just able to pass some given Turing test? I feel like there's some conflation about what lookup table we're talking about. Also, my reading of that passage by Aaronson is very different from yours. I read it as him saying, these computer scientists have put forward a fairly serious and compelling set of arguments that the actual complexity class of the translator algorithm has philosophical importance. Meanwhile, the analytical philosophy response is just to say something hand-wavy about "meaning." I agree Aaronson's not saying anything definitive (he is extremely conservative about making definitive claims, despite how hotly involved he becomes in the few cases when he does make definitive claims). But I don't agree that he is framing it to raise questions about the validity of the CS rebuttal papers that he cites.

dbpokorny 11y ago

In theory we could rig up a quad copter with a machine gun, loudspeaker, and AI, with programming that demands that every human it encounters be interrogated with the question, "do I, a mechanical computer, have the ability to think?" and if the human gives a negative reply, it uses the machine gun on the human until the human is dead or decides to utter a positive response: "yes, this flying death machine can think". Whether or not we say "machines can think" is a political question, and political power comes out of the barrel of a gun. If a machine can wield political power, then it can get you to say "machines can think" because truth is inherently political.

cousin_it 11y ago
You can also be forced to say that 2+2=5, but that doesn't make it true. "Can machines think?" demands a substantive answer.
- dbpokorny 11y ago
  The answer to that question depends on whether or not a robot is pointing a gun at your head. Might makes right.
  - AnimalMuppet 11y ago
    Might can make people lie. That does not make the lies into the truth.
    - dbpokorny 11y ago
      I'm not qualified to continue this discussion further. https://www.youtube.com/watch?v=GicWZl9HmU4 https://www.youtube.com/watch?v=GicWZl9HmU4 https://en.wikipedia.org/wiki/Two_truths_doctrine https://en.wikipedia.org/wiki/Two_truths_doctrine

makebelieve 11y ago

Searle is making an argument about awareness. That a computer system is explicitly unaware of any of it's content. That it's programs are functions and the data also performs a purely functional role. In essence, computers cannot engage in acts of meaning. The programmers and users are engaged in acts of meaning. For instance, saying a program "has a bug", is a completely misleading statement. No programs have bugs. It is impossible for a program to have a bug, just as it is impossible for a physical process to "do something wrong". Programs do what they do, just as molecular processes do what they do. The concept of error and meaning does not exist in a program, just as it does not exist in the physical universe. Meaning (and errors and bugs are a kind of meaning) are things outside programs and outside physics. When a program "has a bug" it means the programmer screwed up, not the program. A program cannot produce errors, because programs, and computer systems in general, do not have the capacity to have meaning. This is what Searle is demonstrating with his argument. This is true for all the popular computational approaches we have today. However, because the human brain appears to function in a purely physical way, and computers function in a purely physical way, it should be theoretically possible to create a computer system that is conscious and aware of meaning just as we are. You refer to this as "Strong AI". Other refer to it as Artificial General Intelligence. I refer to this as machine consciousness. To solve the machine consciousness problem means understanding how awareness, meaning, and representation in general, works. Then building a computer system that engages in representation and instantiates awareness. If an actual person were put into Searle's box, the person would learn chinese. Also, the person could 'intentionally' produce incorrect answers annoying the "programmers" who set the box up in the first place. But a modern computer system cannot 'intentionally' produce errors. it's completely non-sensical to talk about computers as having intention at all. programmers have intention, not computers. Solving the intentionality problem is the other leg of machine consciousness. Elon Musk, Steven Hawking, Nick Bostrom and others make arguments about the dangers of an AI (of any variety) which may acquire intentionality and representational ability, while ignoring the actual deep problems embedded in acquiring those abilities. Awareness, representation, and intention are so fundamental to experience that we have a very difficult time understanding when they happen and when they do not. We see a representational world all around us, but very explicitly, there are no representations at all in the physical world. I believe machine consciousness is possible, but none of the existing approaches will get us there. Searle's chinese room is one succinct argument as to why. The approach I am taking is a kind of metabolic computing. Where single function processes interact in some way similar to molecular interactions and those processes are developed to produce, computational structures like membranes and DNA and eventually "cells". These cells then form multi-cellular structures. These multi-cellular structures and underlying "molecular" interactions instantiate representations and representational processes, like a nervous system. A computational nervous system which embodies representation, intention, sensation, action, imagination, and because it engages in representation making, would be aware. I would love to hear someone describe how any kind of computational approach can produce meaning inside a computer system. We produce meaning and representations so easily; it's hard to understand the difference of perspective necessary to see how representations must form. If someone has an easier approach than the one I am taking, I would be very interested in seeing how they solve the problems of meaning and intention with code.

cousin_it 11y ago

Some people are uncomfortable with the idea that brains can work on "mere" physics. But what's the alternative? A decree from Cthulhu saying human brains are the only devices in the universe that can communicate with consciousness, and all other objects are doomed to be dumb matter? I think that's really unlikely. So it seems like we must bite the bullet and admit that the computation implemented in the Chinese room exhibits genuine understanding. I'm not completely certain, because we don't really understand consciousness yet and we could always find new evidence, but the evidence so far points in one direction.

fallingfrog 11y ago

Well, a Chinese room can simulate a computer, a computer can run a physics simulation and a brain is a physical thing and brains are conscious. So it's pretty open and shut imho; yes, a Chinese room can be conscious. That much is clear. Now what it means to say that something is conscious is another question. I've never seen the word rigorously defined.

chrisdevereux 11y ago

Its popularity in undergraduate philosophy courses reflects the fact that it is easy to criticise more than it telling us something interesting about consciousness.

makebelieve 11y ago

you might want to watch this lecture by Searle at Google: https://www.youtube.com/watch?v=rHKwIYsPXLg https://www.youtube.com/watch?v=rHKwIYsPXLg

wsieroci 11y ago
I saw it and I am constantly pretty amazed just how people can't grasp his argument.