8 ms·

This is one of those topics that Searle wants folks to think straight about. He'd say: You are conscious. You are also a computer. You are not, as far as we ca

This is one of those topics that Searle wants folks to think straight about. He'd say: You are conscious. You are also a computer. You are not, as far as we can tell, conscious by virtue of being a computer. They're just two things that happen to be true about you but they are not linked in that super-direct way. Similarly, you can think. And you can generate text. But you are not, as far as we can tell, thinking by virtue of generating text.

the_gipsy 3y ago
How can we tell? All I hear in my head is some monologue, or hypothetical dialogues with or between other persons. Sounds a lot like predicting text.
- cj 3y ago
  I think very few people think one word at a time, like a LLM. A lot of people think in concepts without words at all in many scenarios. E.g. when you cook an egg, do you reason (in words) with yourself to decide what temperature to set the dial on the stove? Or do you just do it “without really thinking”? Does your brain respond with annoying disclaimers when asking yourself medical questions? Does your brain refuse to entertain an idea if it’s questionable?
  - jncfhnb 3y ago
    LLMs “think” in vector spaces, and “operate” by picking out words one at a time. A full “idea” is there from the beginning. If you say “choose X or Y and then give three supporting arguments” the LLM does not write out the supporting arguments, but the vector space determining the initial one word answer of X or Y does include the embedded awareness of those arguments to different levels of specificity and relevance. The annoying disclaimers and refusals are not fundamental to LLMs but specific LLM services.
  - jddj 3y ago
    Observe your thought process while reading the following sentences. If you're not with us you're... If you scratch my back I'll.. A bird in the hand is worth... What is a bird in the hand worth? How did you know that?
    - danShumway 3y ago
      I'm not sure what this is intended to demonstrate. Your response to "not all problems are solved through generating text" is "but what if the problem is specifically to generate text?" Sure? Yes, if you ask me to complete a sentence, I complete a sentence. That doesn't indicate that human brains are reducible to text generators. If anything, what's interesting about your example is that it might actually in some ways demonstrate the opposite of your point. How many instances are there of people memorizing text or song lyrics and singing along with an artist dozens of times, and only later sitting down and thinking about what the words actually mean? For humans, it's well understood that memorization and repetition of a text is not necessarily the same thing as understanding the concepts within that text. I feel confident I could find you some kids that know that "a bird in the hand is worth two in the bush" that have never actually thought about what that phrase is intending to convey.
      - jddj 3y ago
        I didn't ask you to complete a sentence, though. Just to read them. Then I asked a question. I think you've made my point quite well.
        danShumway 3y ago
        I'm going to be completely honest, I still don't know what your point is. That humans are capable of memorizing text? Is that a thing anyone was debating?
        jddj 3y ago
        Drawing a line around the bits that aren't language generation unfairly ignores the massive apparatus that seems to be specialised (oftentimes overly so) at language generation. Sure there's other stuff going on, but a lot of it is language and a lot of that subset seems to be quite closely approximated by these larger LLMs, warts and all.
  - fragmede 3y ago
    [flagged]
    - cj 3y ago
      If my mind worked like a LLM I would generate some BS reply. Instead, I’ll respond by saying “I don’t know” what you mean. Who knew the ability (or inability) to say “I don’t know” would be so critical to intelligence until now.
      - fragmede 3y ago
        When I asked ChatGPT what "jdnrirmd-gurlksjd" is, it says it doesn't recognize it and needs more context, but is readily able to tell me about the Sapir-Whorf Hypothesis. https://chat.openai.com/share/4f36fe14-0201-4a69-aaf5-2e2e2994398f https://chat.openai.com/share/4f36fe14-0201-4a69-aaf5-2e2e29... When asked what happened last week, ChatGPT will be honest and tell me that its training data only goes up until Jan 2022, and that it doesn't know anything after that date. https://chat.openai.com/share/ca5bd2e1-e91c-468c-9594-be3ac811be04 https://chat.openai.com/share/ca5bd2e1-e91c-468c-9594-be3ac8...
        cj 3y ago
        Except, for your first example I would have said “that’s gibberish” instead of asking for more context since I know more context wouldn’t help. Edit: how did you get GPT to only output bullet points? If the answer is you told it to, why do you have to tell it every time? Don’t you think it would be smart enough to get the hint about your preferences? I’ll stop now though. The debate about whether people think like LLM’s is silly. We clearly don’t, and it doesn’t make sense why people want us to think we do. Edit 2: it’s also frustrating that the “I don’t know” example you’re using was hard coded by OpenAI and not generated by the LLM itself.
        fragmede 3y ago
        Unfortunately the chat link doesn't show the custom instructions that I've set. User message: > Be terse. Do not offer unprompted advice or clarifications. > Remain neutral on all topics. > Never apologize. Model message: > Outline key points
- danShumway 3y ago
  There is debate over whether or not every person has an internal monologue, and a nontrivial number of people self-report that they don't experience conscious dialogs while they think: https://www.verywellmind.com/does-everyone-have-an-inner-monologue-6831748 https://www.verywellmind.com/does-everyone-have-an-inner-mon... It's certainly possible that they're wrong, but my suspicion is that at the very least the form their internal monologue is taking is not one of consciously simulating hypothetical conversations in their head. Additionally, we know that human decisions can in some situations be influenced by stimuli that take effect before the brain has even consciously registered that a decision needs to be made. It seems pretty safe to say that those stimuli are not being processed using a language model.
  - nomel 3y ago
    > It seems pretty safe to say that those stimuli are not being processed using a language model. Could you expand on that? I don’t see how it follows. If there is unconscious output, why wouldn’t it suggest there’s something analogous to a “unthinking” language model stuck behind our conscious, self reflective, bits? How does unconscious output prove that it’s not something similar to a language model? Or, are you speaking on a technicality, saying it’s not a literal language model? I think the idea is something similar and, obviously, multi modal, not a literal LLM/text predictor implementation like runs on GPUs.
    - danShumway 3y ago
      Short answer, the responses are happening too fast for that stimuli to be going through the language centers of our brain. In order to say that the processing of that stimuli is like a language model, we would need to widen our definition of a language model to encompass basically any communication between neurons happening within the brain. ---- So on one hand, you could call that a technicality, on the other hand I would say that if we reduce "like an LLM" to mean "responds to a stimuli by generating a response", at that point "like an LLM" is so broad as to be meaningless. Sure, we can define "humans use a language model" as "human brains generate signals to other parts of the human brain in response to stimuli" -- but that's not really a useful thing to say when we're comparing human reasoning to GPT-4. If we're talking so broadly about reasoning, then it would be just as accurate to say that human brains are like Markov chains. After all, Markov chains interpret inputs and transform them into other signals that they send to other parts of the chain. But if I told you that GPT-4 was just a Markov chain you would (rightfully) disagree with me on that and you would (rightfully) say that I was oversimplifying what's going on within GPT-4's model. Typically, when people say that human beings are text generators, they're talking in the context of GPT-4; they're trying to claim that GPT-4's behavior is analogous to human behavior. Whether or not GPT-4 can reason under any definition of reasoning (direct analogy to human thought should not be the only way we think about reasoning), the way GPT works and the way that it's trained and taught is extremely alien to how human brains work and how humans learn. Broadening the definition doesn't change the fact that LLMs and human brains are very different in practical ways that influence their observable behaviors. ---- I don't think people are consciously trying to do this, but the debate over whether humans are LLMs often ends up feeling like an attempt to simultaneously broaden and narrow a definition at the same time. What people seem to want to do is to describe LLMs extremely broadly but then assume that conclusions they draw from that broad definition are necessarily applicable to a very narrow definition of an LLM (specifically, GPT and similar models). But once we define an LLM broadly enough to encompass the entirety of human reasoning, we have also broadened the category enough to encompass a lot of processes that no one would call reasoning. What doesn't transform inputs into other forms and pass them along to another object? An AC/DC electrical converter does that, but that doesn't mean it can think. It's like claiming, "computers are made of carbon and so are humans, so computers are basically the same as humans." Well, wait a second -- the category as you've defined it is so broad that objects within that category can no longer be assumed to share other attributes with each other.
- pineaux 3y ago
  I dont think in words at all. That is why -quite often- I know what i want to tell but the correct word just doesnt pop up in my head. I can then explain the concept and other people can help me find the word in question. Even while writing this little response, this happened. You see in the first sentence: I know there is a nice word that could be used instead of "doesn't pop up" it resembles the word "evades"? You know when something just keeps out of your grasp, that word... my mind does a little movie and my "mouthbrain" tries to find the nice words for it. That is how I perceive it to be. I dont know how other people perceive this. I imagine they have little words popping up or something. But I dont understand how that kind of thinking could be used to think and reason. How can relationships between concepts be intuitively understood if one does not have a mental picture?
  - the_gipsy 3y ago
    Do "relationships between concepts" have any meaning, for us humans, if we can't express them?
    - cj 3y ago
      Humans can express them. But we rarely do, outside of (maybe) therapy.
- deleted 3y ago
  [deleted]
bondarchuk 3y ago
But Searle also believes obviously wrong things like the Chinese room argument.
- crdrost 3y ago
  The Chinese room argument is just missing kinda one basic premise and people don't realize it and argue around the premise without actually addressing it. “Can we agree that of the many different things folks mean when they say your consciousness is ‘causally generated by your brain,’ that at least we can say your brain in its present state is only capable of generating exactly one conscious process at a time, namely you?” So it's a sort of causal sufficiency, Searle believes that there is something about this squishy machine that the physics is doing that makes it conscious, and that's being caused very directly by the squishy machine. It's not some magic. On the other hand, if you believed in souls, you might think that a given brain contained both the soul of a human person, and the soul of some demon possessing them—two conscious processes, one of which was stuck “along for the ride.” [Searle is also interested in the cases of split-brain patients where you have a similar phenomenon because essentially two separate brains inhabit a single body. He has I think mentioned at a talk that one of the interesting things about consciousness to him is that it all gets Unified, so it's interesting that if the two parts of the brain can talk to each other they merge their consciousnesses into one more powerful consciousness, rather like (my analogy, not his) how if you have two water droplets on a plastic plate and you push them with a toothpick together, at some point they merge into one bigger drop.] Now the Chinese room thought experiment is not about the computational model of consciousness—not directly! It was always phrased as a rebuttal of the Turing test in particular, and the computational model only indirectly after the Turing test falls. Note that the Turing test has no direct beef with the casual sufficiency axiom, which is why it went unstated originally. According to the Turing test, written text goes in, written text comes back out, a dialogue appears to happen to the outsider, this is sufficient to conclude conscious understanding of the language used, which confirms consciousness. Searle’s objection is, “if it's really all about inputs and outputs and not how I get it done, then you've left out what for me feels like the most important part about understanding a language: understanding it is part of how I get it done!” Right? You understand English because you can phrase your ideas into it, you can mold it to suit you, and it can (when heard) so mold you—it’s not just because the words can come out of your mouth, triggered by other words that came in through your ears. The Turing test has always ever stated “don't worry what happens in the middle” and Searle is saying “but for me understanding is part of the details that are happening in the middle!” So where does the computational theory come in? It comes in because Searle wants to make this argument rigorous! He says, “if your computational theory is true, then there is in fact another way that I can speak a language—words in, words out—where I don't understand a word of Chinese. I am Turing complete, I could memorize a program for speaking Chinese and happen to execute it flawlessly and at no point would my ideas get into Chinese, at no point would the things that I heard mold me. So the Turing test is crap!” The prominence of the words “I, me” is what invokes the idea of causal sufficiency here, “I have the right inputs and outputs, but I don't understand.” The Turing test is not able to distinguish between multiple consciousnesses instantiated by the same hardware, if such is even possible. The inputs and outputs go into the same box, as far as Turing is concerned as long as only one conversation happens, there's only one person in there. This does do a great job of defanging the Turing test, because all of the ways that you might weaken the concept to address this major limitation do make it sound completely tautological. “Yeah well something consciousnessy is happening in that box but I don't know what.” / “Okay then why are we even talking about it.” / “Because computers can speak!” / “Right, so we care that computers can speak because they can speak?” / “No, like, we gotta give them rights now, or some shit.” / “John Searle already has human rights, if he's memorized a program that lets him speak Chinese without understanding it, you're saying we need to give that program human rights?” / “Yeah!” / “So uh is it murder if Searle decides that running the program is no longer fun? Is he a slave to this program forever?” / “Uhhhh...” It doesn't defang the computational model, not directly. But the computational model does imply that VMs exist. We use them every day! And that's all that the Chinese room is, it's running a VM inside of another computer, one consciousness carrying a separate consciousness inside of it, a willful sort of demonic possession. The only thing the Chinese room has to say about this, is that we don't use our language very well if it is true. Philosophers who believe in that will need to generate an alternative language that is able to distinguish between “I am doing it” and “I am sustaining a daemon who is doing it,” because for them that's a real difference, you might have a hundred consciousnesses in your head that you don't have direct access to. That is a necessary part of believing that consciousness is software, you don't know if you're in a VM inside your brain, you don't know if something you're doing is actually secretly a Brainfuck program instantiating another VM inside your consciousness, software embeds within software, that's a core feature of software. But of course Searle thinks that that's kind of ridiculous because he thinks that it's obvious that consciousness is something that the squishy wetware of the brain does, and this forces him to believe in that causal sufficiency—“my brain is only sustaining one consciousness, namely me,”—which the computationalists cannot ever agree with because that's not how software works. Anyone who believes in that causal sufficiency, even if they don't have the same basis that Searle does for believing in it, also thinks that the computationalists are ridiculous. But the point is, that's happening at a level way above the Chinese room argument, Chinese room baseball came and went, now this is a whole separate game being played at the same ballpark afterwards.

the_gipsy 3y ago

How can we tell? All I hear in my head is some monologue, or hypothetical dialogues with or between other persons. Sounds a lot like predicting text.

cj 3y ago
I think very few people think one word at a time, like a LLM. A lot of people think in concepts without words at all in many scenarios. E.g. when you cook an egg, do you reason (in words) with yourself to decide what temperature to set the dial on the stove? Or do you just do it “without really thinking”? Does your brain respond with annoying disclaimers when asking yourself medical questions? Does your brain refuse to entertain an idea if it’s questionable?
- jncfhnb 3y ago
  LLMs “think” in vector spaces, and “operate” by picking out words one at a time. A full “idea” is there from the beginning. If you say “choose X or Y and then give three supporting arguments” the LLM does not write out the supporting arguments, but the vector space determining the initial one word answer of X or Y does include the embedded awareness of those arguments to different levels of specificity and relevance. The annoying disclaimers and refusals are not fundamental to LLMs but specific LLM services.
- jddj 3y ago
  Observe your thought process while reading the following sentences. If you're not with us you're... If you scratch my back I'll.. A bird in the hand is worth... What is a bird in the hand worth? How did you know that?
  - danShumway 3y ago
    I'm not sure what this is intended to demonstrate. Your response to "not all problems are solved through generating text" is "but what if the problem is specifically to generate text?" Sure? Yes, if you ask me to complete a sentence, I complete a sentence. That doesn't indicate that human brains are reducible to text generators. If anything, what's interesting about your example is that it might actually in some ways demonstrate the opposite of your point. How many instances are there of people memorizing text or song lyrics and singing along with an artist dozens of times, and only later sitting down and thinking about what the words actually mean? For humans, it's well understood that memorization and repetition of a text is not necessarily the same thing as understanding the concepts within that text. I feel confident I could find you some kids that know that "a bird in the hand is worth two in the bush" that have never actually thought about what that phrase is intending to convey.
    - jddj 3y ago
      I didn't ask you to complete a sentence, though. Just to read them. Then I asked a question. I think you've made my point quite well.
      - danShumway 3y ago
        I'm going to be completely honest, I still don't know what your point is. That humans are capable of memorizing text? Is that a thing anyone was debating?
        jddj 3y ago
        Drawing a line around the bits that aren't language generation unfairly ignores the massive apparatus that seems to be specialised (oftentimes overly so) at language generation. Sure there's other stuff going on, but a lot of it is language and a lot of that subset seems to be quite closely approximated by these larger LLMs, warts and all.
- fragmede 3y ago
  [flagged]
  - cj 3y ago
    If my mind worked like a LLM I would generate some BS reply. Instead, I’ll respond by saying “I don’t know” what you mean. Who knew the ability (or inability) to say “I don’t know” would be so critical to intelligence until now.
    - fragmede 3y ago
      When I asked ChatGPT what "jdnrirmd-gurlksjd" is, it says it doesn't recognize it and needs more context, but is readily able to tell me about the Sapir-Whorf Hypothesis. https://chat.openai.com/share/4f36fe14-0201-4a69-aaf5-2e2e2994398f https://chat.openai.com/share/4f36fe14-0201-4a69-aaf5-2e2e29... When asked what happened last week, ChatGPT will be honest and tell me that its training data only goes up until Jan 2022, and that it doesn't know anything after that date. https://chat.openai.com/share/ca5bd2e1-e91c-468c-9594-be3ac811be04 https://chat.openai.com/share/ca5bd2e1-e91c-468c-9594-be3ac8...
      - cj 3y ago
        Except, for your first example I would have said “that’s gibberish” instead of asking for more context since I know more context wouldn’t help. Edit: how did you get GPT to only output bullet points? If the answer is you told it to, why do you have to tell it every time? Don’t you think it would be smart enough to get the hint about your preferences? I’ll stop now though. The debate about whether people think like LLM’s is silly. We clearly don’t, and it doesn’t make sense why people want us to think we do. Edit 2: it’s also frustrating that the “I don’t know” example you’re using was hard coded by OpenAI and not generated by the LLM itself.
        fragmede 3y ago
        Unfortunately the chat link doesn't show the custom instructions that I've set. User message: > Be terse. Do not offer unprompted advice or clarifications. > Remain neutral on all topics. > Never apologize. Model message: > Outline key points
danShumway 3y ago
There is debate over whether or not every person has an internal monologue, and a nontrivial number of people self-report that they don't experience conscious dialogs while they think: https://www.verywellmind.com/does-everyone-have-an-inner-monologue-6831748 https://www.verywellmind.com/does-everyone-have-an-inner-mon... It's certainly possible that they're wrong, but my suspicion is that at the very least the form their internal monologue is taking is not one of consciously simulating hypothetical conversations in their head. Additionally, we know that human decisions can in some situations be influenced by stimuli that take effect before the brain has even consciously registered that a decision needs to be made. It seems pretty safe to say that those stimuli are not being processed using a language model.
- nomel 3y ago
  > It seems pretty safe to say that those stimuli are not being processed using a language model. Could you expand on that? I don’t see how it follows. If there is unconscious output, why wouldn’t it suggest there’s something analogous to a “unthinking” language model stuck behind our conscious, self reflective, bits? How does unconscious output prove that it’s not something similar to a language model? Or, are you speaking on a technicality, saying it’s not a literal language model? I think the idea is something similar and, obviously, multi modal, not a literal LLM/text predictor implementation like runs on GPUs.
  - danShumway 3y ago
    Short answer, the responses are happening too fast for that stimuli to be going through the language centers of our brain. In order to say that the processing of that stimuli is like a language model, we would need to widen our definition of a language model to encompass basically any communication between neurons happening within the brain. ---- So on one hand, you could call that a technicality, on the other hand I would say that if we reduce "like an LLM" to mean "responds to a stimuli by generating a response", at that point "like an LLM" is so broad as to be meaningless. Sure, we can define "humans use a language model" as "human brains generate signals to other parts of the human brain in response to stimuli" -- but that's not really a useful thing to say when we're comparing human reasoning to GPT-4. If we're talking so broadly about reasoning, then it would be just as accurate to say that human brains are like Markov chains. After all, Markov chains interpret inputs and transform them into other signals that they send to other parts of the chain. But if I told you that GPT-4 was just a Markov chain you would (rightfully) disagree with me on that and you would (rightfully) say that I was oversimplifying what's going on within GPT-4's model. Typically, when people say that human beings are text generators, they're talking in the context of GPT-4; they're trying to claim that GPT-4's behavior is analogous to human behavior. Whether or not GPT-4 can reason under any definition of reasoning (direct analogy to human thought should not be the only way we think about reasoning), the way GPT works and the way that it's trained and taught is extremely alien to how human brains work and how humans learn. Broadening the definition doesn't change the fact that LLMs and human brains are very different in practical ways that influence their observable behaviors. ---- I don't think people are consciously trying to do this, but the debate over whether humans are LLMs often ends up feeling like an attempt to simultaneously broaden and narrow a definition at the same time. What people seem to want to do is to describe LLMs extremely broadly but then assume that conclusions they draw from that broad definition are necessarily applicable to a very narrow definition of an LLM (specifically, GPT and similar models). But once we define an LLM broadly enough to encompass the entirety of human reasoning, we have also broadened the category enough to encompass a lot of processes that no one would call reasoning. What doesn't transform inputs into other forms and pass them along to another object? An AC/DC electrical converter does that, but that doesn't mean it can think. It's like claiming, "computers are made of carbon and so are humans, so computers are basically the same as humans." Well, wait a second -- the category as you've defined it is so broad that objects within that category can no longer be assumed to share other attributes with each other.
pineaux 3y ago
I dont think in words at all. That is why -quite often- I know what i want to tell but the correct word just doesnt pop up in my head. I can then explain the concept and other people can help me find the word in question. Even while writing this little response, this happened. You see in the first sentence: I know there is a nice word that could be used instead of "doesn't pop up" it resembles the word "evades"? You know when something just keeps out of your grasp, that word... my mind does a little movie and my "mouthbrain" tries to find the nice words for it. That is how I perceive it to be. I dont know how other people perceive this. I imagine they have little words popping up or something. But I dont understand how that kind of thinking could be used to think and reason. How can relationships between concepts be intuitively understood if one does not have a mental picture?
- the_gipsy 3y ago
  Do "relationships between concepts" have any meaning, for us humans, if we can't express them?
  - cj 3y ago
    Humans can express them. But we rarely do, outside of (maybe) therapy.
deleted 3y ago
[deleted]

bondarchuk 3y ago

But Searle also believes obviously wrong things like the Chinese room argument.

crdrost 3y ago
The Chinese room argument is just missing kinda one basic premise and people don't realize it and argue around the premise without actually addressing it. “Can we agree that of the many different things folks mean when they say your consciousness is ‘causally generated by your brain,’ that at least we can say your brain in its present state is only capable of generating exactly one conscious process at a time, namely you?” So it's a sort of causal sufficiency, Searle believes that there is something about this squishy machine that the physics is doing that makes it conscious, and that's being caused very directly by the squishy machine. It's not some magic. On the other hand, if you believed in souls, you might think that a given brain contained both the soul of a human person, and the soul of some demon possessing them—two conscious processes, one of which was stuck “along for the ride.” [Searle is also interested in the cases of split-brain patients where you have a similar phenomenon because essentially two separate brains inhabit a single body. He has I think mentioned at a talk that one of the interesting things about consciousness to him is that it all gets Unified, so it's interesting that if the two parts of the brain can talk to each other they merge their consciousnesses into one more powerful consciousness, rather like (my analogy, not his) how if you have two water droplets on a plastic plate and you push them with a toothpick together, at some point they merge into one bigger drop.] Now the Chinese room thought experiment is not about the computational model of consciousness—not directly! It was always phrased as a rebuttal of the Turing test in particular, and the computational model only indirectly after the Turing test falls. Note that the Turing test has no direct beef with the casual sufficiency axiom, which is why it went unstated originally. According to the Turing test, written text goes in, written text comes back out, a dialogue appears to happen to the outsider, this is sufficient to conclude conscious understanding of the language used, which confirms consciousness. Searle’s objection is, “if it's really all about inputs and outputs and not how I get it done, then you've left out what for me feels like the most important part about understanding a language: understanding it is part of how I get it done!” Right? You understand English because you can phrase your ideas into it, you can mold it to suit you, and it can (when heard) so mold you—it’s not just because the words can come out of your mouth, triggered by other words that came in through your ears. The Turing test has always ever stated “don't worry what happens in the middle” and Searle is saying “but for me understanding is part of the details that are happening in the middle!” So where does the computational theory come in? It comes in because Searle wants to make this argument rigorous! He says, “if your computational theory is true, then there is in fact another way that I can speak a language—words in, words out—where I don't understand a word of Chinese. I am Turing complete, I could memorize a program for speaking Chinese and happen to execute it flawlessly and at no point would my ideas get into Chinese, at no point would the things that I heard mold me. So the Turing test is crap!” The prominence of the words “I, me” is what invokes the idea of causal sufficiency here, “I have the right inputs and outputs, but I don't understand.” The Turing test is not able to distinguish between multiple consciousnesses instantiated by the same hardware, if such is even possible. The inputs and outputs go into the same box, as far as Turing is concerned as long as only one conversation happens, there's only one person in there. This does do a great job of defanging the Turing test, because all of the ways that you might weaken the concept to address this major limitation do make it sound completely tautological. “Yeah well something consciousnessy is happening in that box but I don't know what.” / “Okay then why are we even talking about it.” / “Because computers can speak!” / “Right, so we care that computers can speak because they can speak?” / “No, like, we gotta give them rights now, or some shit.” / “John Searle already has human rights, if he's memorized a program that lets him speak Chinese without understanding it, you're saying we need to give that program human rights?” / “Yeah!” / “So uh is it murder if Searle decides that running the program is no longer fun? Is he a slave to this program forever?” / “Uhhhh...” It doesn't defang the computational model, not directly. But the computational model does imply that VMs exist. We use them every day! And that's all that the Chinese room is, it's running a VM inside of another computer, one consciousness carrying a separate consciousness inside of it, a willful sort of demonic possession. The only thing the Chinese room has to say about this, is that we don't use our language very well if it is true. Philosophers who believe in that will need to generate an alternative language that is able to distinguish between “I am doing it” and “I am sustaining a daemon who is doing it,” because for them that's a real difference, you might have a hundred consciousnesses in your head that you don't have direct access to. That is a necessary part of believing that consciousness is software, you don't know if you're in a VM inside your brain, you don't know if something you're doing is actually secretly a Brainfuck program instantiating another VM inside your consciousness, software embeds within software, that's a core feature of software. But of course Searle thinks that that's kind of ridiculous because he thinks that it's obvious that consciousness is something that the squishy wetware of the brain does, and this forces him to believe in that causal sufficiency—“my brain is only sustaining one consciousness, namely me,”—which the computationalists cannot ever agree with because that's not how software works. Anyone who believes in that causal sufficiency, even if they don't have the same basis that Searle does for believing in it, also thinks that the computationalists are ridiculous. But the point is, that's happening at a level way above the Chinese room argument, Chinese room baseball came and went, now this is a whole separate game being played at the same ballpark afterwards.