4 ms·

The chinese room thought experiment is myopic. It focuses on a philosophical distinction that may not actually exist in reality (the concept, and perhaps the il

The chinese room thought experiment is myopic. It focuses on a philosophical distinction that may not actually exist in reality (the concept, and perhaps the illusion, of understanding). In terms of danger, thoughts and feelings are irrelevant. The only thing that matters is agency and action -- and a mimic which guesses and acts out what a sentient entity might do is exactly as dangerous as the sentient entity itself. Waxing philosophical about the nature of cognition is entirely beside the point.

agravier 4y ago
It matters to understand how things work in order to understand their behaviour and react properly. I've seen people draw conclusions from applying Theory of Mind tests to LLMs (https://arxiv.org/abs/2302.02083 https://arxiv.org/abs/2302.02083). Those psychological were designed to assess humans psychological abilities or deficiencies, they assume that the language used by the human respondent reflects their continued and deep understanding of others' state of mind. In LLMs, there is no understanding involved. Dismissing the Chinese Room argument is an ostrich strategy. You're refusing to consider its lack of understanding despite knowing pretty well how an LLM work, because you don't want to ascribe understanding to humans, I suppose?
- throwaway09223 4y ago
  Theory of mind is substantially different from the Chinese Room argument. Theory of mind relates to an ability to predict the responses of another entity/system. An LLM is specifically designed to predict responses. In contrast, the Chinese Room argument is essentially a slight of hand fallacy, shifting "understanding" into a layer of abstraction. It describes a scenario where the human's "understanding of Chinese" is dependent on an external system. It then incorrectly asserts that the human "doesn't understand Chinese" when in fact the union of the human and the human's tools clearly does understand Chinese. In other words, it's fundamentally based around an improper definition of the term "understanding," as well as improper scoping of what constitutes an entity capable of reasoning (the human, vs the human and their tools viewed as a single system). It smacks of a bias of human exceptionalism. It's also guilty of begging the question. The argument attempts to determine the difference between literally understanding Chinese and simulating an understanding -- without addressing whether the two are in fact synonymous. There is no evidence that the human brain isn't also a predictive system.
  - notahacker 4y ago
    The responses to the Chinese Room experiment always seem to involve far more tortuous definition-shifting than the original thought experiment The human in the room understands how to find a list of possible responses to the token 你好吗, and how select a response like 很好 from the list and display that as a response But he human does not understand that 很好 represents an assertion that he is feeling good[1], even though the human has an acute sense of when he feels good or not. He may, in fact, not be feeling particularly good (because, for example he's stuck in a windowless room all day moving strange foreign symbols around!) and have answered completely differently had the question been asked in a language he understood. The books also have no concept of well-being because they're ink on paper. We're really torturing the concept of "understanding" to death to argue that the understanding of a Chinese person who is experiencing 很好 feelings or does not want to admit they actually feel 不好 is indistinguishable from the "understanding" of "the union" of a person who is not feeling 很好 and does not know what 很好 means and some books which do not feel anything contain references to the possibility of replying with 很好, or maybe for variation 好得很, or 不好 which leads to a whole different set of continuations. And the idea that understanding of how you're feeling - the sentiment conveyed to the interlocutor in Chinese - is synonymous with knowing which bookshelf to find continuations where 很好 has been invoked is far too ludicrous to need addressing. The only other relevant entity is the Chinese speaker who designed the room, who would likely have a deep appreciation of feeling 很好, 好得很 and 不好 as well as the appropriate use of those words he designed into the system, but Searle's argument wasn't that programmers weren't sentient. [1]and ironically, I also don't speak Chinese and have relatively little idea what senses 很好 means "good" in and how that overlaps with the English concept, beyond understanding that it's an appropriate response to a common greeting which maps to "how are you"
    - throwaway09223 4y ago
      "But he human does not understand that 很好 represents an assertion that he is feeling good" This is an argument about depth and nuance. A speaker can know: a) The response fits (observe people say it) b) Why the response fits, superficially (很 means "very" and 好 means "good") c) The subtext of the response, both superficially and academically (Chinese people don't actually talk like this in most contexts, it's like saying "how do you do?". The response "very good" is a direct translation of English social norms and is also inappropriate for native Chinese culture. The subtext strongly indicates a non-native speaker with a poor colloquial grasp of the language. Understanding the radicals, etymology and cultural history of each character, related nuance: should the response be a play on 好's radicals of mother/child? etc etc) The depth of c is neigh unlimited. People with an exceptionally strong ability in this area are called poets. It is possible to simulate all of these things. LLMs are surprisingly good at tone and subtext, and are ever improving in these predictive areas. Importantly: While the translating human may not agree or embody the meaning or subtext of the translation. I say "I'm fine" with I'm not fine literally all the time. It's extremely common for humans alone to say things they don't agree with, and for humans alone to express things that they don't fully understand. For a great example of this, consider psychoanalysis: An entire field of practice in large part dedicated to helping people understand what they really mean when they say things (Why did you say you're fine when you're not fine? Let's talk about your choices ...). It is extremely common for human beings to go through the motions of communication without being truly aware of what exactly they're communicating, and why. In fact, no one has a complete grasp of category "C". Particular disabilities can draw these types of limited awareness and mimicry by humans into extremely sharp contrast. "And the idea that understanding of how you're feeling - the sentiment conveyed to the interlocutor in Chinese - is synonymous with knowing which bookshelf to find continuations where 很好 has been invoked is far too ludicrous to need addressing." I don't agree. It's not ludicrous, and as LLMs show it's merely an issue of having a bookshelf of sufficient size and complexity. That's the entire point! Furthermore, this kind of pattern matching is probably how the majority of uneducated people actually communicate. The majority of human beings are reactive. It's our natural state. Mindful, thoughtful communications are a product of intensive training and education and even then a significant portion of human communications are relatively thoughtless. It is a fallacy to assume otherwise. It is also a fallacy to assume that human brains are a single reasoning entity, when it's well established that this is not how brains operate. Freud introduced the rider and horse model for cognition a century ago, and more recent discoveries underscore that the brain cannot be reasonably viewed as a single cohesive thought producing entity. Humans act and react for all sorts of reasons. Finally, it is a fallacy to assume that humans aren't often parroting language that they've seen others use without understanding what it means. This is extremely common, for example people who learn phrases or definitions incorrectly because humans learn language largely by inference. Sometimes we infer incorrectly and for all "intensive purposes" this is the same dynamic -- if you'll pardon the exemplary pun. In a discussion around the nature of cognition and understanding as it applies to tools it makes no sense whatsoever to introduce a hybrid human/tool scenario and then fail to address that the combined system of a human and their tool might be considered to have an understanding, even if the small part of the brain dealing with what we call consciousness doesn't incorporate all of that information directly. "[1]and ironically, I also don't speak Chinese " Ironically I do speak Chinese, although at a fairly basic level (HSK2-3 or so). I've studied fairly casually for about three years. Almost no one says 你好 in real life, though appropriate greetings can be region specific. You might instead to a friend say 你吃了吗?
standardly 4y ago
> If you leave chatGPT alone what does it do? Nothing. It responds to prompts and that is it. Just defending the OP, he stated ChatGPT does nothing but respond the prompts, which is true. That's not waxing philosophical about the nature of cognition. You sort of latched onto his last sentence and set up a strawman against his overall point. Maybe you didn't mean to, but yeah.
- throwaway09223 4y ago
  You may have missed parts of their comment, specifically "It doesn't have interests, thoughts and feelings," and referring to the Chinese Room argument which is specifically an argument regarding the philosophical nature of cognition.

agravier 4y ago

It matters to understand how things work in order to understand their behaviour and react properly. I've seen people draw conclusions from applying Theory of Mind tests to LLMs (https://arxiv.org/abs/2302.02083 https://arxiv.org/abs/2302.02083). Those psychological were designed to assess humans psychological abilities or deficiencies, they assume that the language used by the human respondent reflects their continued and deep understanding of others' state of mind. In LLMs, there is no understanding involved. Dismissing the Chinese Room argument is an ostrich strategy. You're refusing to consider its lack of understanding despite knowing pretty well how an LLM work, because you don't want to ascribe understanding to humans, I suppose?

throwaway09223 4y ago
Theory of mind is substantially different from the Chinese Room argument. Theory of mind relates to an ability to predict the responses of another entity/system. An LLM is specifically designed to predict responses. In contrast, the Chinese Room argument is essentially a slight of hand fallacy, shifting "understanding" into a layer of abstraction. It describes a scenario where the human's "understanding of Chinese" is dependent on an external system. It then incorrectly asserts that the human "doesn't understand Chinese" when in fact the union of the human and the human's tools clearly does understand Chinese. In other words, it's fundamentally based around an improper definition of the term "understanding," as well as improper scoping of what constitutes an entity capable of reasoning (the human, vs the human and their tools viewed as a single system). It smacks of a bias of human exceptionalism. It's also guilty of begging the question. The argument attempts to determine the difference between literally understanding Chinese and simulating an understanding -- without addressing whether the two are in fact synonymous. There is no evidence that the human brain isn't also a predictive system.
- notahacker 4y ago
  The responses to the Chinese Room experiment always seem to involve far more tortuous definition-shifting than the original thought experiment The human in the room understands how to find a list of possible responses to the token 你好吗, and how select a response like 很好 from the list and display that as a response But he human does not understand that 很好 represents an assertion that he is feeling good[1], even though the human has an acute sense of when he feels good or not. He may, in fact, not be feeling particularly good (because, for example he's stuck in a windowless room all day moving strange foreign symbols around!) and have answered completely differently had the question been asked in a language he understood. The books also have no concept of well-being because they're ink on paper. We're really torturing the concept of "understanding" to death to argue that the understanding of a Chinese person who is experiencing 很好 feelings or does not want to admit they actually feel 不好 is indistinguishable from the "understanding" of "the union" of a person who is not feeling 很好 and does not know what 很好 means and some books which do not feel anything contain references to the possibility of replying with 很好, or maybe for variation 好得很, or 不好 which leads to a whole different set of continuations. And the idea that understanding of how you're feeling - the sentiment conveyed to the interlocutor in Chinese - is synonymous with knowing which bookshelf to find continuations where 很好 has been invoked is far too ludicrous to need addressing. The only other relevant entity is the Chinese speaker who designed the room, who would likely have a deep appreciation of feeling 很好, 好得很 and 不好 as well as the appropriate use of those words he designed into the system, but Searle's argument wasn't that programmers weren't sentient. [1]and ironically, I also don't speak Chinese and have relatively little idea what senses 很好 means "good" in and how that overlaps with the English concept, beyond understanding that it's an appropriate response to a common greeting which maps to "how are you"
  - throwaway09223 4y ago
    "But he human does not understand that 很好 represents an assertion that he is feeling good" This is an argument about depth and nuance. A speaker can know: a) The response fits (observe people say it) b) Why the response fits, superficially (很 means "very" and 好 means "good") c) The subtext of the response, both superficially and academically (Chinese people don't actually talk like this in most contexts, it's like saying "how do you do?". The response "very good" is a direct translation of English social norms and is also inappropriate for native Chinese culture. The subtext strongly indicates a non-native speaker with a poor colloquial grasp of the language. Understanding the radicals, etymology and cultural history of each character, related nuance: should the response be a play on 好's radicals of mother/child? etc etc) The depth of c is neigh unlimited. People with an exceptionally strong ability in this area are called poets. It is possible to simulate all of these things. LLMs are surprisingly good at tone and subtext, and are ever improving in these predictive areas. Importantly: While the translating human may not agree or embody the meaning or subtext of the translation. I say "I'm fine" with I'm not fine literally all the time. It's extremely common for humans alone to say things they don't agree with, and for humans alone to express things that they don't fully understand. For a great example of this, consider psychoanalysis: An entire field of practice in large part dedicated to helping people understand what they really mean when they say things (Why did you say you're fine when you're not fine? Let's talk about your choices ...). It is extremely common for human beings to go through the motions of communication without being truly aware of what exactly they're communicating, and why. In fact, no one has a complete grasp of category "C". Particular disabilities can draw these types of limited awareness and mimicry by humans into extremely sharp contrast. "And the idea that understanding of how you're feeling - the sentiment conveyed to the interlocutor in Chinese - is synonymous with knowing which bookshelf to find continuations where 很好 has been invoked is far too ludicrous to need addressing." I don't agree. It's not ludicrous, and as LLMs show it's merely an issue of having a bookshelf of sufficient size and complexity. That's the entire point! Furthermore, this kind of pattern matching is probably how the majority of uneducated people actually communicate. The majority of human beings are reactive. It's our natural state. Mindful, thoughtful communications are a product of intensive training and education and even then a significant portion of human communications are relatively thoughtless. It is a fallacy to assume otherwise. It is also a fallacy to assume that human brains are a single reasoning entity, when it's well established that this is not how brains operate. Freud introduced the rider and horse model for cognition a century ago, and more recent discoveries underscore that the brain cannot be reasonably viewed as a single cohesive thought producing entity. Humans act and react for all sorts of reasons. Finally, it is a fallacy to assume that humans aren't often parroting language that they've seen others use without understanding what it means. This is extremely common, for example people who learn phrases or definitions incorrectly because humans learn language largely by inference. Sometimes we infer incorrectly and for all "intensive purposes" this is the same dynamic -- if you'll pardon the exemplary pun. In a discussion around the nature of cognition and understanding as it applies to tools it makes no sense whatsoever to introduce a hybrid human/tool scenario and then fail to address that the combined system of a human and their tool might be considered to have an understanding, even if the small part of the brain dealing with what we call consciousness doesn't incorporate all of that information directly. "[1]and ironically, I also don't speak Chinese " Ironically I do speak Chinese, although at a fairly basic level (HSK2-3 or so). I've studied fairly casually for about three years. Almost no one says 你好 in real life, though appropriate greetings can be region specific. You might instead to a friend say 你吃了吗?

standardly 4y ago

> If you leave chatGPT alone what does it do? Nothing. It responds to prompts and that is it. Just defending the OP, he stated ChatGPT does nothing but respond the prompts, which is true. That's not waxing philosophical about the nature of cognition. You sort of latched onto his last sentence and set up a strawman against his overall point. Maybe you didn't mean to, but yeah.

throwaway09223 4y ago
You may have missed parts of their comment, specifically "It doesn't have interests, thoughts and feelings," and referring to the Chinese Room argument which is specifically an argument regarding the philosophical nature of cognition.