6 ms·

The argument is against this view of consciousness. More intuitively: our words are things which do things in the world. When we acquire them we do so by being

The argument is against this view of consciousness. More intuitively: our words are things which do things in the world. When we acquire them we do so by being embedded in the world. When we use them we use them to change the world. (Largely). If a child in a cupboard reading a textbook out-loud could fool a chemist that the child understood chemistry, it doesn't mean they do. In the end, you need to turn on that bunsen burner. The chinese room needs to be taken super-literally: does the person just doing a "hash-table lookup" on "the right answers to pre-prosed questions" actually understand. To me its incredibly obvious that they do not. This does not establish that, eg. cognition, does not involve computational hastable-like processes. I think it does establish that propontents of a naive computationalism have an incredibly big challenge. The challenge is to specify the environment computationally, since that is the only means by which the computer itself can be sufficiently embedded and actually understand anything. The issue here is that if "computer" isnt just a trivial property it has to include, eg., measurability, determinism, etc. Properties that reality lacks. Reality isnt a computer. Starting with the chinese room one /can/ supplement sufficiently until we reach searle's conclusion.

mananaysiempre 5y ago
... I’m finding it very difficult to phrase this argument about actually understanding in a way that wouldn’t also entail that chairs made of atoms (and—overwhelmingly—empty space) don’t actually exist, they are mere simulacra of real chairs (which don’t exist anywhere). Which might be a consistent position, but I’d instead say that it is simply a largely useless interpretation of the word actually and as good philosophers we should search for a better one. (See also Anderson’s “More is different”.) Or is there such a phrasing?
- mjburgess 5y ago
  Approximately, You "actually understand" if in saying words, w, about a situation, s, the reason, r, you say those words counts as as an understanding of s. So if you're asked, "what is the weather today?" and a system replies, "its sunny" -- it only understands if the reason it replied was because its saying "sunny" had come about because it understands sunny sitations; and therefore has a justfiied reason on this occasion to say that this occasion is sunny. If I ask a NN trained on a trillion documents, "do you like what I'm wearing?" it cannot answer. It can say the words, "yes I do!" but it cannot have a reason to say those words. It's just a "weighted average retrievla across a compressed dataset of a trillion documetns". The question isnt "on average, what -- historically -- would a generic person reply to the question: do you like what i'm wearing?" The question is if the systems likes what I'm wearing. Its replies are only "actual replies" if it can see what i'm wearing, has experiences/preferences for taste, has some aesthetic judgement, has a disposition to like/dislike -- etc. No ML system on the planet, in this sense, has any understanding whatsoever. Interpolation over historical data is just a means of compressing history into parameters, ie., its a compressed lookup table. That isn't ever a /reason/, exactly, if a person simply used such a table -- they wouldnt mean what they said
  - vlowther 5y ago
    Given that your squishy brainmeats also consist of neural networks that have been trained directly by years of experience and indirectly by billions of years of evolution, I can write off your response as just a weighted average retrieval across a compressed dataset of trillions of your experiences.
    - mjburgess 5y ago
      If, on the occasion I say, "I like you" my saying it is caused by my liking you -- then you can describe this process however you wish. Since my liking you is caused by my immediate environment, it isn't reducible to a weighted average of my history. Another way of putting it: the historical positions of all the molecules in some water aren't sufficient to determine its present state. It's state depends on its container (ie., the pressure & temp of its environment). And there are a very very large number of states of water, many still being discovered. In this sense my state in any moment is a point in an infinite space of states -- not determined by my history. But also extremely complexly by my container -- my social, etc. environment. The world hitting my senses is doing more to me than the air on the water. It induces in me a state which cannot be "averaged" from my history. Thus, no, we are not weighed averages of our histories. We are profoundly chaotic and organic organisms whose growth in our environments enables us to respond to our enviroments by entering a near infinite number of states. These states arent in our history, they are how our biophysical structure -- via history -- responses to the near infinite depth of the here-and-now. We are more like water than a computer. A computer is a deterministic machine which is a deterministic function of its deterministic inputs. Water is a chatoic system whose state "isnt up to it". Water's state is /in/ its container, and water itself is a non-determinsitic chatoic soup. The chaos of water is the least of what one nanometer of a cell has; a cell is a trillion times that adaptive and responsive. And we are a trillion of those. We are a cascade of chaotic state changes provoked by an infinitely rich environment action on our bodies shaped by a long history of organic growth. We don't have "neural networks", we have cells. That some form "networks" has nothing to do with what we are. A complete misdirection.
      - mannykannot 5y ago
hackinthebochs 5y ago
> does the person just doing a "hash-table lookup" But this is Searle's sleight-of-hand. The question isn't whether the person in the Chinese room understands Chinese, but whether the system as a whole understands Chinese. The man is analogous to the CPU in a computer, but the CPU is embedded in a larger system and it is the entire system that performs operations of the computer. You can't focus on one component to the exclusion of the rest of the system and expect to derive valid conclusions about the system. >The challenge is to specify the environment computationally I agree, but this presents no in principle challenge to a computational system reaching such a detailed description of the environment. >The issue here is that if "computer" isnt just a trivial property it has to include, eg., measurability, determinism, etc. Properties that reality lacks. Reality isnt a computer. I don't follow. The issues of measurability and determinism in QM (assuming that's what you are referring to) aren't necessarily problems at the scale of physical systems. Computational systems can certainly make distinctions to within some margin of error, which is sufficient as a substrate for gathering information about the external environment.
- mjburgess 5y ago
  No, there's no sleight of hand. It seems you just agree that the man doesnt understand Chinese. > at the scale of physical systems. Alas, they are. Even classical mechanics isn't deterministic. Most systems are chaotic and require infinite precision in measurement to be deterministic. Since QM precludes this, most chaotic systems literally can only be predicted over short horizons. Eg., there is moon in our solar system, i believe, with a chaotic orbit. At maximum possible measurement fidelity we can predict 20yrs. Consider that all the particles of our body are just these little chaotic moons, and their aggregate chaos dramatically dimishes this 20yr horizon. There is, quite literally, not enough information present in this moment to predict the next moment. Most systems are sufficiently chaotic in a sufficiently large number of parameters to preclude this. This reaches all the way up to classical scales. Reality isnt a computer.
  - burkaman 5y ago
    Everyone agrees that the man doesn't understand Chinese. The sleight of hand is that Searle compares a full computer to a component of the Chinese Room, when he should be comparing a computer to the room as a whole. The man in this thought experiment is analogous to an internal component of a computer, not the whole thing. Put another way, Searle separates a "computer" from the instructions it is running, but there is no such separation. It's like separating a brain from the electrical brain activity. It's true that a brain without electrical activity cannot understand Chinese just like a computer without a program cannot, but that's not a very interesting observation.
3np 5y ago
> The chinese room needs to be taken super-literally: does the person just doing a "hash-table lookup" on "the right answers to pre-prosed questions" actually understand. That’s not how it’s phrased. The Chinese Room needs to pass the Turing Test with arbitrary input of Chinese characters. So a convincing Chinese Elissa basically. > The issue here is that if "computer" isnt just a trivial property it has to include, eg., measurability, determinism, etc. Properties that reality lacks. Reality isnt a computer. Reality isn’t a human brain either. Or is it? Either way I don’t see how that is relevant.
- denton-scratch 5y ago
  There's a confusion in the language I'm seeing in the comments, between "understand" as in knowing a language, and "understand" as in possessing a useful mental model of a system. I don't see why a machine can't be said to "know" a language; after all, it can translate languages. I can translate from French to English (I'm a native English speaker); but there are sentences in English that I can't "understand", either because I don't really know what all the words mean, or because the concept expressed by the sentence is beyond me.
burkaman 5y ago
> does the person just doing a "hash-table lookup" on "the right answers to pre-prosed questions" actually understand. The person doesn't, but the (person + hash-table) system does. This is not weird - presumably whoever wrote the hash-table did understand Chinese. The whole point of using a room for this thought experiment is that we're no longer talking about a person, we're talking about a "machine" consisting of a person, a hash-table, and some kind of input/output system. The hypothetical computer is the same. The CPU does not understand Chinese, but the whole computer as a system does, because part of that system is actual knowledge that came from someone that understands Chinese. When people ask if a computer is intelligent, they are not asking about one particular component, they're asking about the computer as a whole. Just like when you talk about a human's knowledge and abilities, you don't ask about particular sectors of their brain, you ask about the person as a whole.
- mensetmanusman 5y ago
  The person is frivolous in the Chinese room though, would you say (robot + hash table) understands Chinese? I.e. GTP-3 understands English…
  - burkaman 5y ago
    GPT-3 isn't there yet, but yes, a robot with the book from the Chinese Room experiment would form a system that understands Chinese. I agree that the person is frivolous, that's why it's strange that Searle asks whether the person in his thought experiment understands Chinese. That's irrelevant, what matters is whether the room as a whole does, and clearly it does.
  - deleted 5y ago
    [deleted]
- colinmhayes 5y ago
  I'm not sure. Say the person has translations tables for english and chinese. If the second tester asked in English "what did you talk about with the last person?" would the man be able to answer? Clearly the man + hash table speaks chinese, but I don't think that's the same as understanding it.
mannykannot 5y ago
I agree that one should take Searle's argument literally, and I doubt there are many people on either side who suppose that the room's operator understands either the questions being posed to the room or the answers it delivers. The dispute is whether there is anything significant in this assumption - and, more specifically, whether Searle is right in supposing that if Hard AI is possible, then it would follow that the room's operator must understand the questions, their answers, and why they are appropriate answers. You may recognize that, by distinguishing between the operator and the room, I am prefiguring the so-called "systems reply", which is that the system as a whole is demonstrating whatever level of understanding is manifest here, and there is no assumption, in so supposing, that this understanding is manifest in any single component, even if one of those components has the ability to understand some things on its own. By deliberate construction, Searle is using the room's operator to mechanistically implement an algorithm, a task that does not require knowledge of the language in which the questions are posed (in fact, he is using the language barrier to prevent the operator answering the questions himself.) Searle has not presented an argument for the operator needing to do anything more, and it is obvious that people can be trained to perform tasks they do not have any understanding of beyond the specific actions involved. Searle's attempt to refute this reply seems to show that he cannot even conceive it fully, as that response is to modify the scenario such that the operator memorizes the rule book, as if where it is stored makes any difference. Searle's stance here seems to me to be a version of the homunculus fallacy, supposing that our minds are implemented by (as opposed to being) a conscious entity.
- mjburgess 5y ago
  I think it is significant that the room, even as a system, does not use language. A hashtable lookup cannot be what it is to "act in the world with words". When I say, "do you like what I'm wearing?" -- regardless of what has been written in any book, at any time, I do not want that reply. I may want those words. But the reply I want is to use your judgement of taste and experiences of the world to tell me your opinion. The words dont answer the question if the reason they are used is incorrect. A hashtable lookup can, basically, never be a reason to use words. And hence the Chinese room reveals much much more than simply a circular point. And the systems reply fails to say much much less than it needs to. The systems reply needs to explain "by what system" the system's use of language counts as a use. "by what system" words are born of understanding, not "mere reterival". To do this, you need to specify the whole world, experience, etc. as "computational systems". And thus the systems reply simply fails against this example. It isnt sufficient to say "maybe" you have to say, "and this is how!". And worse, the world is clearly not a computer -- as any non-trivial definition (eg., universal turing machine) cannot simulate properties the world has (non-determinism, chaos, etc.).
dragonwriter 5y ago
> The chinese room needs to be taken super-literally: does the person just doing a "hash-table lookup" on "the right answers to pre-prosed questions" actually understand. > To me its incredibly obvious that they do not. That is Searle’s approach, but I think the point where Searle and others taking it fall down in the inability to do this: Explain any testable manner in which we can confirm that those who you would describe as “really understanding” a language (or anything else) are actually doing something materially different from what you assert is obviously not understanding. Until someone can do this, it is clear that the Chinese Room is nothing more than a conclusion that mechanistic computation is insufficient for understanding resting solely on the assumption that...mechanistic computation is insufficient for understanding.

mananaysiempre 5y ago

... I’m finding it very difficult to phrase this argument about actually understanding in a way that wouldn’t also entail that chairs made of atoms (and—overwhelmingly—empty space) don’t actually exist, they are mere simulacra of real chairs (which don’t exist anywhere). Which might be a consistent position, but I’d instead say that it is simply a largely useless interpretation of the word actually and as good philosophers we should search for a better one. (See also Anderson’s “More is different”.) Or is there such a phrasing?

mjburgess 5y ago
Approximately, You "actually understand" if in saying words, w, about a situation, s, the reason, r, you say those words counts as as an understanding of s. So if you're asked, "what is the weather today?" and a system replies, "its sunny" -- it only understands if the reason it replied was because its saying "sunny" had come about because it understands sunny sitations; and therefore has a justfiied reason on this occasion to say that this occasion is sunny. If I ask a NN trained on a trillion documents, "do you like what I'm wearing?" it cannot answer. It can say the words, "yes I do!" but it cannot have a reason to say those words. It's just a "weighted average retrievla across a compressed dataset of a trillion documetns". The question isnt "on average, what -- historically -- would a generic person reply to the question: do you like what i'm wearing?" The question is if the systems likes what I'm wearing. Its replies are only "actual replies" if it can see what i'm wearing, has experiences/preferences for taste, has some aesthetic judgement, has a disposition to like/dislike -- etc. No ML system on the planet, in this sense, has any understanding whatsoever. Interpolation over historical data is just a means of compressing history into parameters, ie., its a compressed lookup table. That isn't ever a /reason/, exactly, if a person simply used such a table -- they wouldnt mean what they said
- vlowther 5y ago
  Given that your squishy brainmeats also consist of neural networks that have been trained directly by years of experience and indirectly by billions of years of evolution, I can write off your response as just a weighted average retrieval across a compressed dataset of trillions of your experiences.
  - mjburgess 5y ago
    If, on the occasion I say, "I like you" my saying it is caused by my liking you -- then you can describe this process however you wish. Since my liking you is caused by my immediate environment, it isn't reducible to a weighted average of my history. Another way of putting it: the historical positions of all the molecules in some water aren't sufficient to determine its present state. It's state depends on its container (ie., the pressure & temp of its environment). And there are a very very large number of states of water, many still being discovered. In this sense my state in any moment is a point in an infinite space of states -- not determined by my history. But also extremely complexly by my container -- my social, etc. environment. The world hitting my senses is doing more to me than the air on the water. It induces in me a state which cannot be "averaged" from my history. Thus, no, we are not weighed averages of our histories. We are profoundly chaotic and organic organisms whose growth in our environments enables us to respond to our enviroments by entering a near infinite number of states. These states arent in our history, they are how our biophysical structure -- via history -- responses to the near infinite depth of the here-and-now. We are more like water than a computer. A computer is a deterministic machine which is a deterministic function of its deterministic inputs. Water is a chatoic system whose state "isnt up to it". Water's state is /in/ its container, and water itself is a non-determinsitic chatoic soup. The chaos of water is the least of what one nanometer of a cell has; a cell is a trillion times that adaptive and responsive. And we are a trillion of those. We are a cascade of chaotic state changes provoked by an infinitely rich environment action on our bodies shaped by a long history of organic growth. We don't have "neural networks", we have cells. That some form "networks" has nothing to do with what we are. A complete misdirection.
    - mannykannot 5y ago

hackinthebochs 5y ago

> does the person just doing a "hash-table lookup" But this is Searle's sleight-of-hand. The question isn't whether the person in the Chinese room understands Chinese, but whether the system as a whole understands Chinese. The man is analogous to the CPU in a computer, but the CPU is embedded in a larger system and it is the entire system that performs operations of the computer. You can't focus on one component to the exclusion of the rest of the system and expect to derive valid conclusions about the system. >The challenge is to specify the environment computationally I agree, but this presents no in principle challenge to a computational system reaching such a detailed description of the environment. >The issue here is that if "computer" isnt just a trivial property it has to include, eg., measurability, determinism, etc. Properties that reality lacks. Reality isnt a computer. I don't follow. The issues of measurability and determinism in QM (assuming that's what you are referring to) aren't necessarily problems at the scale of physical systems. Computational systems can certainly make distinctions to within some margin of error, which is sufficient as a substrate for gathering information about the external environment.

mjburgess 5y ago
No, there's no sleight of hand. It seems you just agree that the man doesnt understand Chinese. > at the scale of physical systems. Alas, they are. Even classical mechanics isn't deterministic. Most systems are chaotic and require infinite precision in measurement to be deterministic. Since QM precludes this, most chaotic systems literally can only be predicted over short horizons. Eg., there is moon in our solar system, i believe, with a chaotic orbit. At maximum possible measurement fidelity we can predict 20yrs. Consider that all the particles of our body are just these little chaotic moons, and their aggregate chaos dramatically dimishes this 20yr horizon. There is, quite literally, not enough information present in this moment to predict the next moment. Most systems are sufficiently chaotic in a sufficiently large number of parameters to preclude this. This reaches all the way up to classical scales. Reality isnt a computer.
- burkaman 5y ago
  Everyone agrees that the man doesn't understand Chinese. The sleight of hand is that Searle compares a full computer to a component of the Chinese Room, when he should be comparing a computer to the room as a whole. The man in this thought experiment is analogous to an internal component of a computer, not the whole thing. Put another way, Searle separates a "computer" from the instructions it is running, but there is no such separation. It's like separating a brain from the electrical brain activity. It's true that a brain without electrical activity cannot understand Chinese just like a computer without a program cannot, but that's not a very interesting observation.

3np 5y ago

> The chinese room needs to be taken super-literally: does the person just doing a "hash-table lookup" on "the right answers to pre-prosed questions" actually understand. That’s not how it’s phrased. The Chinese Room needs to pass the Turing Test with arbitrary input of Chinese characters. So a convincing Chinese Elissa basically. > The issue here is that if "computer" isnt just a trivial property it has to include, eg., measurability, determinism, etc. Properties that reality lacks. Reality isnt a computer. Reality isn’t a human brain either. Or is it? Either way I don’t see how that is relevant.

denton-scratch 5y ago
There's a confusion in the language I'm seeing in the comments, between "understand" as in knowing a language, and "understand" as in possessing a useful mental model of a system. I don't see why a machine can't be said to "know" a language; after all, it can translate languages. I can translate from French to English (I'm a native English speaker); but there are sentences in English that I can't "understand", either because I don't really know what all the words mean, or because the concept expressed by the sentence is beyond me.

burkaman 5y ago

> does the person just doing a "hash-table lookup" on "the right answers to pre-prosed questions" actually understand. The person doesn't, but the (person + hash-table) system does. This is not weird - presumably whoever wrote the hash-table did understand Chinese. The whole point of using a room for this thought experiment is that we're no longer talking about a person, we're talking about a "machine" consisting of a person, a hash-table, and some kind of input/output system. The hypothetical computer is the same. The CPU does not understand Chinese, but the whole computer as a system does, because part of that system is actual knowledge that came from someone that understands Chinese. When people ask if a computer is intelligent, they are not asking about one particular component, they're asking about the computer as a whole. Just like when you talk about a human's knowledge and abilities, you don't ask about particular sectors of their brain, you ask about the person as a whole.

mensetmanusman 5y ago
The person is frivolous in the Chinese room though, would you say (robot + hash table) understands Chinese? I.e. GTP-3 understands English…
- burkaman 5y ago
  GPT-3 isn't there yet, but yes, a robot with the book from the Chinese Room experiment would form a system that understands Chinese. I agree that the person is frivolous, that's why it's strange that Searle asks whether the person in his thought experiment understands Chinese. That's irrelevant, what matters is whether the room as a whole does, and clearly it does.
- deleted 5y ago
  [deleted]
colinmhayes 5y ago
I'm not sure. Say the person has translations tables for english and chinese. If the second tester asked in English "what did you talk about with the last person?" would the man be able to answer? Clearly the man + hash table speaks chinese, but I don't think that's the same as understanding it.

mannykannot 5y ago

I agree that one should take Searle's argument literally, and I doubt there are many people on either side who suppose that the room's operator understands either the questions being posed to the room or the answers it delivers. The dispute is whether there is anything significant in this assumption - and, more specifically, whether Searle is right in supposing that if Hard AI is possible, then it would follow that the room's operator must understand the questions, their answers, and why they are appropriate answers. You may recognize that, by distinguishing between the operator and the room, I am prefiguring the so-called "systems reply", which is that the system as a whole is demonstrating whatever level of understanding is manifest here, and there is no assumption, in so supposing, that this understanding is manifest in any single component, even if one of those components has the ability to understand some things on its own. By deliberate construction, Searle is using the room's operator to mechanistically implement an algorithm, a task that does not require knowledge of the language in which the questions are posed (in fact, he is using the language barrier to prevent the operator answering the questions himself.) Searle has not presented an argument for the operator needing to do anything more, and it is obvious that people can be trained to perform tasks they do not have any understanding of beyond the specific actions involved. Searle's attempt to refute this reply seems to show that he cannot even conceive it fully, as that response is to modify the scenario such that the operator memorizes the rule book, as if where it is stored makes any difference. Searle's stance here seems to me to be a version of the homunculus fallacy, supposing that our minds are implemented by (as opposed to being) a conscious entity.

mjburgess 5y ago
I think it is significant that the room, even as a system, does not use language. A hashtable lookup cannot be what it is to "act in the world with words". When I say, "do you like what I'm wearing?" -- regardless of what has been written in any book, at any time, I do not want that reply. I may want those words. But the reply I want is to use your judgement of taste and experiences of the world to tell me your opinion. The words dont answer the question if the reason they are used is incorrect. A hashtable lookup can, basically, never be a reason to use words. And hence the Chinese room reveals much much more than simply a circular point. And the systems reply fails to say much much less than it needs to. The systems reply needs to explain "by what system" the system's use of language counts as a use. "by what system" words are born of understanding, not "mere reterival". To do this, you need to specify the whole world, experience, etc. as "computational systems". And thus the systems reply simply fails against this example. It isnt sufficient to say "maybe" you have to say, "and this is how!". And worse, the world is clearly not a computer -- as any non-trivial definition (eg., universal turing machine) cannot simulate properties the world has (non-determinism, chaos, etc.).

dragonwriter 5y ago

> The chinese room needs to be taken super-literally: does the person just doing a "hash-table lookup" on "the right answers to pre-prosed questions" actually understand. > To me its incredibly obvious that they do not. That is Searle’s approach, but I think the point where Searle and others taking it fall down in the inability to do this: Explain any testable manner in which we can confirm that those who you would describe as “really understanding” a language (or anything else) are actually doing something materially different from what you assert is obviously not understanding. Until someone can do this, it is clear that the Chinese Room is nothing more than a conclusion that mechanistic computation is insufficient for understanding resting solely on the assumption that...mechanistic computation is insufficient for understanding.