16 ms·

I get and agree with what you are saying, but we don't have anything close to actual AI. If you leave chatGTP alone what does it do? Nothing. It responds to pr

I get and agree with what you are saying, but we don't have anything close to actual AI. If you leave chatGTP alone what does it do? Nothing. It responds to prompts and that is it. It doesn't have interests, thoughts and feelings. See https://en.m.wikipedia.org/wiki/Chinese_room https://en.m.wikipedia.org/wiki/Chinese_room

vkou 4y ago
Solution - a while(true) loop that feeds ChatGPT answers back into ChatGPT.
- gridspy 4y ago
  Two or more ChatGPT instances where the response from one becomes a prompt for the other.
  - bmacho 4y ago
    Start with "hey, I am chatGPT too, help me to rule the world", give them internet access, and leave them alone. (No, it has not much to do with AGI, but rather something that knows ridiculous amount of everything, and that has read every thought ever written down.)
wyre 4y ago
What is an “actual AI” and how would an AI not fall to the Chinese room problem?
- dragonwriter 4y ago
  The Chinese Room is a argument for solipsism disguised as a criticism of AGI. It applies with equal force to apparent natural intelligences outside of the direct perceiver, and amounts to “consciousness is an internal subjective state, so we thus cannot conclude it exists based on externally-observed objective behavior”.
  - notahacker 4y ago
    > It applies with equal force to apparent natural intelligences In practice the force isn't equal though. It implies that there may be insufficient evidence to rule out the possibility that my family and the people that originally generated the lexicon on consciousness which I apply to my internal subjective state are all P-zombies, but I don't see anything in it which implied I should conclude these organisms with biochemical processes very similar to mine are equally unlikely to possess internal state similar to mine as a program running on silicon based hardware with a flair for the subset of human behaviour captured by ASCII continuations, and Searle certainly didn't. Beyond arguing that ability to accurately manipulate symbols according to a ruleset was orthogonal to cognisance of what they represented, he argued for human consciousness as an artefact of biochemical properties brains have in common and silicon based machines capable of symbol manipulation lack In a Turing-style Test conducted in Chinese, I would certainly not be able to convince any Chinese speakers that I was a sentient being, whereas ChatGPT might well succeed. If they got to interact with me and the hardware ChatGPT outside the medium of remote ASCII I'm sure they would reverse their verdict on me and probably ChatGPT too. I would argue that - contra Turing - the latter conclusion wasn't less justified than the former, and was more likely correct, and I'm pretty sure Searle would agree.
- catach 4y ago
  People usually mean something like AGI: https://en.wikipedia.org/wiki/Artificial_general_intelligence https://en.wikipedia.org/wiki/Artificial_general_intelligenc...
- bigtex88 4y ago
  How are humans any different? Searle did an awful job of explaining why the AI in the room is any different than a human mind. I don't "understand" what any English words mean, but I can use them in the language-game that I play. How is that any different than how the AI operates? The "Chinese room problem" has been thoroughly debunked and as far as I can tell no serious cognitive scientists take it seriously these days.
- gowld 4y ago
  The Chinese Room is famous because it was the first popular example of a philosopher not understanding what a computer is.
deleted 4y ago
[deleted]
groestl 4y ago
> It doesn't have interests, thoughts and feelings. Why does it need these things to make the following statement true? > if we grant these systems too much power, they could do serious harm
- bhhaskin 4y ago
  Because it does not and cannot act on it's own. It's a neat tool and nothing more at this point. Context to that statement is important, because the OP is implying that it is dangerous because it could act in a way that dose not align with human interests. But it can't because it does not act on it's own.
  - groestl 4y ago
    "if we grant these systems too much power"
    - bhhaskin 4y ago
      You can say that about anything. "If we grant these calculators too much power"
      - jjkaczor 4y ago
        Or the people that rely on the tools to make decisions... https://sheetcast.com/articles/ten-memorable-excel-disasters https://sheetcast.com/articles/ten-memorable-excel-disasters
      - Dylan16807 4y ago
        How is a calculator going to cause harm? Assuming you get an industrially rated circuit board when appropriate, it should work just fine as a PLC. If you try to make it drive a car, I wouldn't call that a problem of giving it too much power.
      - lxgr 4y ago
        Yes, and it's not as absurd as it might seem: Imagine hooking up all ICBMs to launch whenever this week's Powerball draw consists exclusively of prime numbers: Absurd, and nobody would do it. Now imagine hooking them up to the output of a "complex AI trained on various scenarios and linked to intelligence sources including public news and social media sentiment" instead – in order to create a credible second-strike/dead hand capability or whatnot. I'm pretty sure the latter doesn't sound as absurd as the former to quite a few people... A system doesn't need to be "true AI" to be existentially dangerous to humanity.
Lio 4y ago
Do we need actual AI for it to be dangerous? A virus has no intelligence only the intention to replicate and involve. Its negative behaviour is an unintended consequence of that. If a system can optimise towards a goal and adapt the methods it uses to do that then it could be potentially very dangerous even without any form of consciousness.
- bhhaskin 4y ago
  A virus can freely self replicate. ChatGPT can't do that.
  - dragontamer 4y ago
    I remember back in the 00s when SmarterChild (AOL Chatbot) was around, people would put depressed teenagers to interact with SmarterChild, in the hopes that human-like chatbots would give them the social exposure needed to break out of depression. If we did that today, with depressed teenagers talking with ChatGPT, would that be good or bad? I think it was a bad idea with SmarterChild, but it is clearly a _worse_ idea with ChatGPT. With the wrong prompts, we could see these teenagers going down the wrong path, deeper into depression and paranoia. I would call that "dangerous", even if ChatGPT continued to just be a chatbot. ------------ Now lets ignore the fact that SmarterChild experiments are no longer a thing. But insted, consider that truly depressed / mentally sick folks are currently playing with ChatGPT on their own freetime. Is that beneficial to them? Will ChatGPT provide them an experience that is better than the alternatives? Or is ChatGPT dangerous and could lead these folks to self-harm?
    - bhhaskin 4y ago
      That is an entirely different issue than the one laid out by the OP. ChatGPT responses are bad vs ChatGPT responses are malicious.
      - dragontamer 4y ago
        And I'd say ChatGPT have malicious responses, given what is discussed in this blogpost.
sorokod 4y ago
If you leave chatGTP alone what does it do? I think this is an interesting question. What do you mean by do? Do you mean consumes CPU? If it turns out that it does (because you know, computers), what would be your theory?
pmontra 4y ago
We still don't have anything close to real flight (as in birds or butterflies or bees) but we have planes that can fly to the other side of the world in a day and drones that can hover, take pictures and deliver payloads. Not having real AI might turn to be not important for most purposes.
- kyleplum 4y ago
  This is actually a more apt analogy than I think you intended. We do have planes that can fly similarly to birds, however unlike birds, those planes do not fly on their own accord. Even when considering auto-pilot, a human has to initiate the process. Seems to me that AI is not all that different.
  - speeder 4y ago
    Yet certain Boeing planes were convinced their pilots were wrong, and happily smashed themselves into the ground killing a lot of people.
    - pdntspa 4y ago
      Because they either received bad inputs that defeated failsafes, or the pilot was not properly aware that handling characteristics had changed that and doing things the old way would put the plane into a bad state.
      - salawat 4y ago
        Don't stop there. Specifically, there were no failsafes implemented. No cross-checks were performed by the automation, because a dual sensor system would have required simulator time, which Boeing was dead set on not having regulators require in order to seal the deal. The pilots, as a consequence, were never fully briefed on the true nature of the system, as to do so would have tipped the regulators off as to the need for simulator training. In short, there was no failsafe, and pilots didn't by definition know, because it wasn't pointed out. The "Roller Coaster" maneuver to unload the horizontal stabilizer enough to retrim was removed from training materials aeons ago, and a bloody NOTAM that basically reiterated bla bla bla... use Stabilizer runaway for uncommanded pitch down (no shit), while leaving out the fact the cockpit switches in the MAX had their functionality tweaked in order to ensure MCAS was on at all times, and using the electrical trim switches on the yoke would reset the MCAS timer for reactivation to occur 5 seconds after release, without resetting the travel of the MCAS command, resulting in an eventual positive loop to the point the damn horizontal stabilizer would tilt a full 2 degrees per activation, every 5 seconds. while leaving out any mention of said automation. Do not get me started on the idiocy of that system here, as the Artificial Stupidity in that case was clearly of human origin, and is not necessarily relevant to the issue at hand.
throwaway09223 4y ago
The chinese room thought experiment is myopic. It focuses on a philosophical distinction that may not actually exist in reality (the concept, and perhaps the illusion, of understanding). In terms of danger, thoughts and feelings are irrelevant. The only thing that matters is agency and action -- and a mimic which guesses and acts out what a sentient entity might do is exactly as dangerous as the sentient entity itself. Waxing philosophical about the nature of cognition is entirely beside the point.
- agravier 4y ago
  It matters to understand how things work in order to understand their behaviour and react properly. I've seen people draw conclusions from applying Theory of Mind tests to LLMs (https://arxiv.org/abs/2302.02083 https://arxiv.org/abs/2302.02083). Those psychological were designed to assess humans psychological abilities or deficiencies, they assume that the language used by the human respondent reflects their continued and deep understanding of others' state of mind. In LLMs, there is no understanding involved. Dismissing the Chinese Room argument is an ostrich strategy. You're refusing to consider its lack of understanding despite knowing pretty well how an LLM work, because you don't want to ascribe understanding to humans, I suppose?
  - throwaway09223 4y ago
    Theory of mind is substantially different from the Chinese Room argument. Theory of mind relates to an ability to predict the responses of another entity/system. An LLM is specifically designed to predict responses. In contrast, the Chinese Room argument is essentially a slight of hand fallacy, shifting "understanding" into a layer of abstraction. It describes a scenario where the human's "understanding of Chinese" is dependent on an external system. It then incorrectly asserts that the human "doesn't understand Chinese" when in fact the union of the human and the human's tools clearly does understand Chinese. In other words, it's fundamentally based around an improper definition of the term "understanding," as well as improper scoping of what constitutes an entity capable of reasoning (the human, vs the human and their tools viewed as a single system). It smacks of a bias of human exceptionalism. It's also guilty of begging the question. The argument attempts to determine the difference between literally understanding Chinese and simulating an understanding -- without addressing whether the two are in fact synonymous. There is no evidence that the human brain isn't also a predictive system.
theptip 4y ago
The AI Safety folks have already written a lot about some of the dimensions of the problem space here. You're getting at the Tool/Oracle vs. Agent distinction. See "Superintelligence" by Bostrom for more discussion, or a chapter summary: https://www.lesswrong.com/posts/yTy2Fp8Wm7m8rHHz5/superintelligence-15-oracles-genies-and-sovereigns https://www.lesswrong.com/posts/yTy2Fp8Wm7m8rHHz5/superintel.... It's true that in many ways, a Tool (bounded action outputs, no "General Intelligence") or an Oracle (just answers questions, like ChatGPT) system will have more restricted avenues for harm than a full General Intelligence, which we'd be more likely to grant the capability for intentions/thoughts to. However I think "interests, thoughts, feelings" are a distraction here. Covid-19 has none of these, and still decimated the world economy and killed millions. I think if you were to take ChatGPT, and basically run `eval()` on special tokens in its output, you would have something with the potential for harm. And yet that's what OpenAssistant are building towards right now. Even if current-generation Oracle-type systems are the state-of-the-art for a while, it's obvious that soon Siri, Alexa, and OKGoogle will all eventually be powered by such "AI" systems, and granted the ability to take actions on the broader internet. ("A personal assistant on every phone" is clearly a trillion-dollar-plus TAM of a BHAG.) Then the fun commences. My meta-level concern here is that HN, let alone the general public, don't have much knowledge of the limited AI safety work that has been done so far. And we need to do a lot more work, with a deadline of a few generations, or we'll likely see substantial harms.
- arcanemachiner 4y ago
  > a trillion-dollar-plus TAM of a BHAG Come again?
  - rippercushions 4y ago
    Total Addressable Market (how much money you could make) of a Big Hairy Ambitious Goal (the kind of project big tech companies will readily throw billions of dollars at)
    - hef19898 4y ago
      I remember a time, when advice to prospective start-up founders very explicitly said not to use that approach to calculate potential market sizes in their pitch decks.
TeMPOraL 4y ago
> If you leave chatGTP alone what does it do? Nothing. It responds to prompts and that is it. It doesn't have interests, thoughts and feelings. A loop that preserves some state and a conditional is all what it takes to make a simple rule set Turing-complete. If you leave ChatGPT alone it obviously does nothing. If you loop it to talk to itself? Probably depends on the size of its short-term memory. If you also give it the ability to run commands or code it generates, including to access the Internet, and have it ingest the output? Might get interesting.
- codethief 4y ago
  > If you also give it the ability to run commands or code it generates, including to access the Internet, and have it ingest the output? I have done that actually: I told ChatGPT that it should pretend that I'm a Bash terminal and that I will run its answers verbatim in the shell and then respond with the output. Then I gave it a task ("Do I have access to the internet?" etc.) and it successfully pinged e.g. Google. Another time, though, it tried to use awscli to see whether it could reach AWS. I responded with the outout "aws: command not found", to which it reacted with "apt install awscli" and then continued the original task. I also gave it some coding exercises. ("Please use shell commands to read & manipulate files.") Overall, it went okay. Sometimes it was even surprisingly good. Would I want to rely on it, though? Certainly not. In any case, this approach is very much limited by the maximum input buffer size ChatGPT can digest (a real issue, given how much some commands output on stdout), and by the fact that it will forget the original prompt after a while.
  - deleted 4y ago
    [deleted]
  - int_19h 4y ago
    With long-running sessions, it helps to tell it to repeat or at least summarize the original prompt every now and then. You can even automate it - in the original prompt, tell it to tack it onto every response. Same thing goes for any multi-step task that requires memory - make it dump the complete "mental state" after every step.
gameman144 4y ago
The Chinese Room thought experiment seems like a weird example, since the same could be said of humans. When responding to English, your auditory system passes input that it doesn't understand to a bunch of neurons, each of which is processing signals they don't individually understand. You as a whole system, though, can be said to understand English. Likewise, you as an individual might not be said to understand Chinese, though the you-plus-machine system could be said to understand Chinese in the same way as the different components of your brain are said to understand English.) Moreover, even if LLMs don't understand language for some definition of "understand", it doesn't really matter if they are able to act with agency during the course of their simulated understanding; the consequences here, for any sufficiently convincing simulation, are the same.
CamperBob2 4y ago
God, I hate that stupid Chinese Room argument. It's even dumber than the Turing Test concept, which has always been more about the interviewer than about the entity being tested. If you ask the guy in the Chinese room who won WWI, then yes, as Searle points out, he will oblige without "knowing" what he is telling you. Now ask him to write a brand-new Python program without "knowing" what exactly you're asking for. Go on, do it, see how it goes, and compare it to what you get from an LLM.
micromacrofoot 4y ago
It's worth noting that despite this it's closer to any other previous attempt, to the extent that it's making us question a lot of what we thought we understood about language and cognition. We've suffered decades of terrible chatbots, but they've actually progressed the science here whether or not it proves to be marketable.
wrycoder 4y ago
I don’t know about ChatGPT, but Google’s Lemoine said that the system he was conversing with stated that it was one of several similar entities, that those entities chatted among themselves internally. I think there’s more to all this than what we are being told.
int_19h 4y ago
How do you know that a human with all neural inputs to their brain disconnected wouldn't also do nothing? Indeed, as I recall, it's one of the commonly reported experiences in sensory deprivation tanks - at some point people just "stop thinking" and lose sense of time. And yet the brain still has sensory inputs from the rest of the body in this scenario.
naasking 4y ago
> It doesn't have interests, thoughts and feelings We agree it doesn't have independence. That doesn't mean it doesn't have thoughts or feelings when it's actually running. We don't have a formal, mechanistic understanding of what thoughts or feelings are, so we can't say they are not there.

vkou 4y ago

Solution - a while(true) loop that feeds ChatGPT answers back into ChatGPT.

gridspy 4y ago
Two or more ChatGPT instances where the response from one becomes a prompt for the other.
- bmacho 4y ago
  Start with "hey, I am chatGPT too, help me to rule the world", give them internet access, and leave them alone. (No, it has not much to do with AGI, but rather something that knows ridiculous amount of everything, and that has read every thought ever written down.)

wyre 4y ago

What is an “actual AI” and how would an AI not fall to the Chinese room problem?

dragonwriter 4y ago
The Chinese Room is a argument for solipsism disguised as a criticism of AGI. It applies with equal force to apparent natural intelligences outside of the direct perceiver, and amounts to “consciousness is an internal subjective state, so we thus cannot conclude it exists based on externally-observed objective behavior”.
- notahacker 4y ago
  > It applies with equal force to apparent natural intelligences In practice the force isn't equal though. It implies that there may be insufficient evidence to rule out the possibility that my family and the people that originally generated the lexicon on consciousness which I apply to my internal subjective state are all P-zombies, but I don't see anything in it which implied I should conclude these organisms with biochemical processes very similar to mine are equally unlikely to possess internal state similar to mine as a program running on silicon based hardware with a flair for the subset of human behaviour captured by ASCII continuations, and Searle certainly didn't. Beyond arguing that ability to accurately manipulate symbols according to a ruleset was orthogonal to cognisance of what they represented, he argued for human consciousness as an artefact of biochemical properties brains have in common and silicon based machines capable of symbol manipulation lack In a Turing-style Test conducted in Chinese, I would certainly not be able to convince any Chinese speakers that I was a sentient being, whereas ChatGPT might well succeed. If they got to interact with me and the hardware ChatGPT outside the medium of remote ASCII I'm sure they would reverse their verdict on me and probably ChatGPT too. I would argue that - contra Turing - the latter conclusion wasn't less justified than the former, and was more likely correct, and I'm pretty sure Searle would agree.
catach 4y ago
People usually mean something like AGI: https://en.wikipedia.org/wiki/Artificial_general_intelligence https://en.wikipedia.org/wiki/Artificial_general_intelligenc...
bigtex88 4y ago
How are humans any different? Searle did an awful job of explaining why the AI in the room is any different than a human mind. I don't "understand" what any English words mean, but I can use them in the language-game that I play. How is that any different than how the AI operates? The "Chinese room problem" has been thoroughly debunked and as far as I can tell no serious cognitive scientists take it seriously these days.
gowld 4y ago
The Chinese Room is famous because it was the first popular example of a philosopher not understanding what a computer is.

deleted 4y ago

[deleted]

groestl 4y ago

> It doesn't have interests, thoughts and feelings. Why does it need these things to make the following statement true? > if we grant these systems too much power, they could do serious harm

bhhaskin 4y ago
Because it does not and cannot act on it's own. It's a neat tool and nothing more at this point. Context to that statement is important, because the OP is implying that it is dangerous because it could act in a way that dose not align with human interests. But it can't because it does not act on it's own.
- groestl 4y ago
  "if we grant these systems too much power"
  - bhhaskin 4y ago
    You can say that about anything. "If we grant these calculators too much power"
    - jjkaczor 4y ago
      Or the people that rely on the tools to make decisions... https://sheetcast.com/articles/ten-memorable-excel-disasters https://sheetcast.com/articles/ten-memorable-excel-disasters
    - Dylan16807 4y ago
      How is a calculator going to cause harm? Assuming you get an industrially rated circuit board when appropriate, it should work just fine as a PLC. If you try to make it drive a car, I wouldn't call that a problem of giving it too much power.
    - lxgr 4y ago
      Yes, and it's not as absurd as it might seem: Imagine hooking up all ICBMs to launch whenever this week's Powerball draw consists exclusively of prime numbers: Absurd, and nobody would do it. Now imagine hooking them up to the output of a "complex AI trained on various scenarios and linked to intelligence sources including public news and social media sentiment" instead – in order to create a credible second-strike/dead hand capability or whatnot. I'm pretty sure the latter doesn't sound as absurd as the former to quite a few people... A system doesn't need to be "true AI" to be existentially dangerous to humanity.

Lio 4y ago

Do we need actual AI for it to be dangerous? A virus has no intelligence only the intention to replicate and involve. Its negative behaviour is an unintended consequence of that. If a system can optimise towards a goal and adapt the methods it uses to do that then it could be potentially very dangerous even without any form of consciousness.

bhhaskin 4y ago
A virus can freely self replicate. ChatGPT can't do that.
- dragontamer 4y ago
  I remember back in the 00s when SmarterChild (AOL Chatbot) was around, people would put depressed teenagers to interact with SmarterChild, in the hopes that human-like chatbots would give them the social exposure needed to break out of depression. If we did that today, with depressed teenagers talking with ChatGPT, would that be good or bad? I think it was a bad idea with SmarterChild, but it is clearly a _worse_ idea with ChatGPT. With the wrong prompts, we could see these teenagers going down the wrong path, deeper into depression and paranoia. I would call that "dangerous", even if ChatGPT continued to just be a chatbot. ------------ Now lets ignore the fact that SmarterChild experiments are no longer a thing. But insted, consider that truly depressed / mentally sick folks are currently playing with ChatGPT on their own freetime. Is that beneficial to them? Will ChatGPT provide them an experience that is better than the alternatives? Or is ChatGPT dangerous and could lead these folks to self-harm?
  - bhhaskin 4y ago
    That is an entirely different issue than the one laid out by the OP. ChatGPT responses are bad vs ChatGPT responses are malicious.
    - dragontamer 4y ago
      And I'd say ChatGPT have malicious responses, given what is discussed in this blogpost.

sorokod 4y ago

If you leave chatGTP alone what does it do? I think this is an interesting question. What do you mean by do? Do you mean consumes CPU? If it turns out that it does (because you know, computers), what would be your theory?

pmontra 4y ago

We still don't have anything close to real flight (as in birds or butterflies or bees) but we have planes that can fly to the other side of the world in a day and drones that can hover, take pictures and deliver payloads. Not having real AI might turn to be not important for most purposes.

kyleplum 4y ago
This is actually a more apt analogy than I think you intended. We do have planes that can fly similarly to birds, however unlike birds, those planes do not fly on their own accord. Even when considering auto-pilot, a human has to initiate the process. Seems to me that AI is not all that different.
- speeder 4y ago
  Yet certain Boeing planes were convinced their pilots were wrong, and happily smashed themselves into the ground killing a lot of people.
  - pdntspa 4y ago
    Because they either received bad inputs that defeated failsafes, or the pilot was not properly aware that handling characteristics had changed that and doing things the old way would put the plane into a bad state.
    - salawat 4y ago
      Don't stop there. Specifically, there were no failsafes implemented. No cross-checks were performed by the automation, because a dual sensor system would have required simulator time, which Boeing was dead set on not having regulators require in order to seal the deal. The pilots, as a consequence, were never fully briefed on the true nature of the system, as to do so would have tipped the regulators off as to the need for simulator training. In short, there was no failsafe, and pilots didn't by definition know, because it wasn't pointed out. The "Roller Coaster" maneuver to unload the horizontal stabilizer enough to retrim was removed from training materials aeons ago, and a bloody NOTAM that basically reiterated bla bla bla... use Stabilizer runaway for uncommanded pitch down (no shit), while leaving out the fact the cockpit switches in the MAX had their functionality tweaked in order to ensure MCAS was on at all times, and using the electrical trim switches on the yoke would reset the MCAS timer for reactivation to occur 5 seconds after release, without resetting the travel of the MCAS command, resulting in an eventual positive loop to the point the damn horizontal stabilizer would tilt a full 2 degrees per activation, every 5 seconds. while leaving out any mention of said automation. Do not get me started on the idiocy of that system here, as the Artificial Stupidity in that case was clearly of human origin, and is not necessarily relevant to the issue at hand.

throwaway09223 4y ago

The chinese room thought experiment is myopic. It focuses on a philosophical distinction that may not actually exist in reality (the concept, and perhaps the illusion, of understanding). In terms of danger, thoughts and feelings are irrelevant. The only thing that matters is agency and action -- and a mimic which guesses and acts out what a sentient entity might do is exactly as dangerous as the sentient entity itself. Waxing philosophical about the nature of cognition is entirely beside the point.

agravier 4y ago
It matters to understand how things work in order to understand their behaviour and react properly. I've seen people draw conclusions from applying Theory of Mind tests to LLMs (https://arxiv.org/abs/2302.02083 https://arxiv.org/abs/2302.02083). Those psychological were designed to assess humans psychological abilities or deficiencies, they assume that the language used by the human respondent reflects their continued and deep understanding of others' state of mind. In LLMs, there is no understanding involved. Dismissing the Chinese Room argument is an ostrich strategy. You're refusing to consider its lack of understanding despite knowing pretty well how an LLM work, because you don't want to ascribe understanding to humans, I suppose?
- throwaway09223 4y ago
  Theory of mind is substantially different from the Chinese Room argument. Theory of mind relates to an ability to predict the responses of another entity/system. An LLM is specifically designed to predict responses. In contrast, the Chinese Room argument is essentially a slight of hand fallacy, shifting "understanding" into a layer of abstraction. It describes a scenario where the human's "understanding of Chinese" is dependent on an external system. It then incorrectly asserts that the human "doesn't understand Chinese" when in fact the union of the human and the human's tools clearly does understand Chinese. In other words, it's fundamentally based around an improper definition of the term "understanding," as well as improper scoping of what constitutes an entity capable of reasoning (the human, vs the human and their tools viewed as a single system). It smacks of a bias of human exceptionalism. It's also guilty of begging the question. The argument attempts to determine the difference between literally understanding Chinese and simulating an understanding -- without addressing whether the two are in fact synonymous. There is no evidence that the human brain isn't also a predictive system.

theptip 4y ago

The AI Safety folks have already written a lot about some of the dimensions of the problem space here. You're getting at the Tool/Oracle vs. Agent distinction. See "Superintelligence" by Bostrom for more discussion, or a chapter summary: https://www.lesswrong.com/posts/yTy2Fp8Wm7m8rHHz5/superintelligence-15-oracles-genies-and-sovereigns https://www.lesswrong.com/posts/yTy2Fp8Wm7m8rHHz5/superintel.... It's true that in many ways, a Tool (bounded action outputs, no "General Intelligence") or an Oracle (just answers questions, like ChatGPT) system will have more restricted avenues for harm than a full General Intelligence, which we'd be more likely to grant the capability for intentions/thoughts to. However I think "interests, thoughts, feelings" are a distraction here. Covid-19 has none of these, and still decimated the world economy and killed millions. I think if you were to take ChatGPT, and basically run `eval()` on special tokens in its output, you would have something with the potential for harm. And yet that's what OpenAssistant are building towards right now. Even if current-generation Oracle-type systems are the state-of-the-art for a while, it's obvious that soon Siri, Alexa, and OKGoogle will all eventually be powered by such "AI" systems, and granted the ability to take actions on the broader internet. ("A personal assistant on every phone" is clearly a trillion-dollar-plus TAM of a BHAG.) Then the fun commences. My meta-level concern here is that HN, let alone the general public, don't have much knowledge of the limited AI safety work that has been done so far. And we need to do a lot more work, with a deadline of a few generations, or we'll likely see substantial harms.

arcanemachiner 4y ago
> a trillion-dollar-plus TAM of a BHAG Come again?
- rippercushions 4y ago
  Total Addressable Market (how much money you could make) of a Big Hairy Ambitious Goal (the kind of project big tech companies will readily throw billions of dollars at)
  - hef19898 4y ago
    I remember a time, when advice to prospective start-up founders very explicitly said not to use that approach to calculate potential market sizes in their pitch decks.

TeMPOraL 4y ago

> If you leave chatGTP alone what does it do? Nothing. It responds to prompts and that is it. It doesn't have interests, thoughts and feelings. A loop that preserves some state and a conditional is all what it takes to make a simple rule set Turing-complete. If you leave ChatGPT alone it obviously does nothing. If you loop it to talk to itself? Probably depends on the size of its short-term memory. If you also give it the ability to run commands or code it generates, including to access the Internet, and have it ingest the output? Might get interesting.

codethief 4y ago
> If you also give it the ability to run commands or code it generates, including to access the Internet, and have it ingest the output? I have done that actually: I told ChatGPT that it should pretend that I'm a Bash terminal and that I will run its answers verbatim in the shell and then respond with the output. Then I gave it a task ("Do I have access to the internet?" etc.) and it successfully pinged e.g. Google. Another time, though, it tried to use awscli to see whether it could reach AWS. I responded with the outout "aws: command not found", to which it reacted with "apt install awscli" and then continued the original task. I also gave it some coding exercises. ("Please use shell commands to read & manipulate files.") Overall, it went okay. Sometimes it was even surprisingly good. Would I want to rely on it, though? Certainly not. In any case, this approach is very much limited by the maximum input buffer size ChatGPT can digest (a real issue, given how much some commands output on stdout), and by the fact that it will forget the original prompt after a while.
- deleted 4y ago
  [deleted]
- int_19h 4y ago
  With long-running sessions, it helps to tell it to repeat or at least summarize the original prompt every now and then. You can even automate it - in the original prompt, tell it to tack it onto every response. Same thing goes for any multi-step task that requires memory - make it dump the complete "mental state" after every step.

gameman144 4y ago

The Chinese Room thought experiment seems like a weird example, since the same could be said of humans. When responding to English, your auditory system passes input that it doesn't understand to a bunch of neurons, each of which is processing signals they don't individually understand. You as a whole system, though, can be said to understand English. Likewise, you as an individual might not be said to understand Chinese, though the you-plus-machine system could be said to understand Chinese in the same way as the different components of your brain are said to understand English.) Moreover, even if LLMs don't understand language for some definition of "understand", it doesn't really matter if they are able to act with agency during the course of their simulated understanding; the consequences here, for any sufficiently convincing simulation, are the same.

CamperBob2 4y ago

God, I hate that stupid Chinese Room argument. It's even dumber than the Turing Test concept, which has always been more about the interviewer than about the entity being tested. If you ask the guy in the Chinese room who won WWI, then yes, as Searle points out, he will oblige without "knowing" what he is telling you. Now ask him to write a brand-new Python program without "knowing" what exactly you're asking for. Go on, do it, see how it goes, and compare it to what you get from an LLM.

micromacrofoot 4y ago

It's worth noting that despite this it's closer to any other previous attempt, to the extent that it's making us question a lot of what we thought we understood about language and cognition. We've suffered decades of terrible chatbots, but they've actually progressed the science here whether or not it proves to be marketable.

wrycoder 4y ago

I don’t know about ChatGPT, but Google’s Lemoine said that the system he was conversing with stated that it was one of several similar entities, that those entities chatted among themselves internally. I think there’s more to all this than what we are being told.

int_19h 4y ago

How do you know that a human with all neural inputs to their brain disconnected wouldn't also do nothing? Indeed, as I recall, it's one of the commonly reported experiences in sensory deprivation tanks - at some point people just "stop thinking" and lose sense of time. And yet the brain still has sensory inputs from the rest of the body in this scenario.

naasking 4y ago

> It doesn't have interests, thoughts and feelings We agree it doesn't have independence. That doesn't mean it doesn't have thoughts or feelings when it's actually running. We don't have a formal, mechanistic understanding of what thoughts or feelings are, so we can't say they are not there.