4 ms·

>which definitely doesn't think Why do you think it "definitely" doesn't think? If this is about lacking consciousness, then you should make that explicit. But

>which definitely doesn't think Why do you think it "definitely" doesn't think? If this is about lacking consciousness, then you should make that explicit. But then the question just gets pushed back further, why are you sure it isn't conscious? The problem with the Chinese room at its core is the problem of attribution. We want to attribute properties like consciousness or understanding to "things" we are familiar with, and the only sufficient thing in the room is the man. But this intuition misdirects us to bad conclusions. The question to ask is what is responding when questions are being asked of the room. The responses are being generated by the algorithm reified into a causally efficacious process. Essentially, the reified algorithm is a set of properties without objecthood. It's so hard for people to see this as a live option because our cognitive makeup is such that we reason based on discrete entities. Considering extant properties without entities to carry them is just an alien notion. We can go further and conceptualize a virtual objecthood for the process to help us reason about the entity we interface with in the Chinese room. We can then ask what properties does it have, how much does it understand, and so on. But we can't simply say it doesn't have any of these properties because it's merely a mechanical/computational process.

seanhunter 4y ago
> Why do you think it "definitely" doesn't think? If this is about lacking consciousness, then you should make that explicit. But then the question just gets pushed back further, why are you sure it isn't conscious? Because we know how GPTs work. They just return a high-probability "most likely next token" in response to what has gone before (the prompt and previous context). For that to be thought we would have to redefine the word.
- hackinthebochs 4y ago
  "Just returning the next token" is an overly reductive description that misses what is powerful about these models. To accurately predict the next token requires the discovery of a model that has a correspondence with the system being modeled to varying degrees of fidelity. In the limit of infinite samples, the model must perfectly capture the structure and information dynamics of the system. Any deficiency in the model will necessarily reveal itself in a difference in output at some point. Of course LLMs aren't trained in the limit of infinite samples, but the question is how much of the system (a human mind producing language) has it modeled? If sentience or understanding is a necessary feature of a mind producing the kinds of texts we produce, it is an open question whether LLMs have or will recover the right sort of structure for these properties.
  - seanhunter 4y ago
    You're trying to pull yourself up by your own bootstraps here. If sentience is required to produce the kinds of texts that LLMs produce then sentience is a feature of LLMs of course. But if sentience is required for my computer to produce the "Tada!" noise when it switches on then my computer is sentient also. (It's probably not sentient.) Nothing you have said establishes in any sense that LLMs are sentient, and the Chinese Room experiment shows that we can't just rely on our intuition.
    - hackinthebochs 4y ago
      I'm not sure what to make of comments like this. The discussion around LLMs and their capacities has just been universally horrendous. >But if sentience is required for my computer to produce the "Tada!" noise when it switches on then my computer is sentient also. But there is a large and obvious difference between your PCs boots up sound process and the output of LLMs. For one, its far more plausible that sentience is required to produce the kinds of texts that LLMs produce than to produce a boot up sound. >Nothing you have said establishes in any sense that LLMs are sentient I wasn't trying to, I was simply pointing out the error in concluding a lack of sentience from the arguments given. If you want to have a discussion about how/why LLMs might be sentient, I'm happy to have it. But most people don't actually want to have that discussion.
- kerpotgh 4y ago
  [dead]

seanhunter 4y ago

> Why do you think it "definitely" doesn't think? If this is about lacking consciousness, then you should make that explicit. But then the question just gets pushed back further, why are you sure it isn't conscious? Because we know how GPTs work. They just return a high-probability "most likely next token" in response to what has gone before (the prompt and previous context). For that to be thought we would have to redefine the word.

hackinthebochs 4y ago
"Just returning the next token" is an overly reductive description that misses what is powerful about these models. To accurately predict the next token requires the discovery of a model that has a correspondence with the system being modeled to varying degrees of fidelity. In the limit of infinite samples, the model must perfectly capture the structure and information dynamics of the system. Any deficiency in the model will necessarily reveal itself in a difference in output at some point. Of course LLMs aren't trained in the limit of infinite samples, but the question is how much of the system (a human mind producing language) has it modeled? If sentience or understanding is a necessary feature of a mind producing the kinds of texts we produce, it is an open question whether LLMs have or will recover the right sort of structure for these properties.
- seanhunter 4y ago
  You're trying to pull yourself up by your own bootstraps here. If sentience is required to produce the kinds of texts that LLMs produce then sentience is a feature of LLMs of course. But if sentience is required for my computer to produce the "Tada!" noise when it switches on then my computer is sentient also. (It's probably not sentient.) Nothing you have said establishes in any sense that LLMs are sentient, and the Chinese Room experiment shows that we can't just rely on our intuition.
  - hackinthebochs 4y ago
    I'm not sure what to make of comments like this. The discussion around LLMs and their capacities has just been universally horrendous. >But if sentience is required for my computer to produce the "Tada!" noise when it switches on then my computer is sentient also. But there is a large and obvious difference between your PCs boots up sound process and the output of LLMs. For one, its far more plausible that sentience is required to produce the kinds of texts that LLMs produce than to produce a boot up sound. >Nothing you have said establishes in any sense that LLMs are sentient I wasn't trying to, I was simply pointing out the error in concluding a lack of sentience from the arguments given. If you want to have a discussion about how/why LLMs might be sentient, I'm happy to have it. But most people don't actually want to have that discussion.
kerpotgh 4y ago
[dead]