6 ms·

ChatGPT seems like it is "just" a very advanced Chinese Room(?). That is, it is optimized for the Turing test: for deceiving human and algorithmic classificatio

ChatGPT seems like it is "just" a very advanced Chinese Room(?). That is, it is optimized for the Turing test: for deceiving human and algorithmic classification to appear to write as "human-like" as possible. Very often this optimization yields "truth-y" or "insightful-ish" responses! But ultimately, this optimization cannot yield a super intelligence, rather it yields a super mockingbird. I don't doubt that a sufficiently complex and well-trained computer program could be super intelligent...I just don't think we'll get there solely by making a computer program try to mimic things that humans have written online (while simultaneously avoiding being identified/classified as a plagiarist or non-human). Edit: I have no idea how I originally called this a "Turing machine". Thank you for correcting my utterly baffling brain error.

satvikpendem 4y ago
Sounds like you're thinking of a Chinese Room. https://en.wikipedia.org/wiki/Chinese_room https://en.wikipedia.org/wiki/Chinese_room
xyzelement 4y ago
Sorry to be the pedant (someone is gonna do it eventually) - a Turing Machine isn't a machine designed to pass the Turing test (what you are actually referring to)
- runnerup 4y ago
  Thank you. I have absolutely no idea how I mixed these up despite being very familiar with all the associated concepts and their respective labels.
  - amelius 4y ago
    Sounds like a mistake GPT could make, suspicious :)
jannes 4y ago
I think you made a mistake in your first sentence. A turing machine is something different: https://en.wikipedia.org/wiki/Turing_machine https://en.wikipedia.org/wiki/Turing_machine
bryan0 4y ago
I think you’re also misunderstanding the point of the Chinese Room. The Chinese room purports to show that strong AI is not possible because all computer programs cannot “understand”. I think this argument falls apart unless you can explain why our brains are not computable.
- runnerup 4y ago
  Fair enough. I hope I was clear that I do think that computer programs can understand. I just don't think that optimizing for solving the Turing test will generally yield a computer program capable of "understanding" or "super intelligence".
  - chii 4y ago
    A turing test is meant to demonstrate one aspect of understanding/thinking. I mean, what else is a good criteria for understanding?
    - mannykannot 4y ago
      Whether, and to what extent, a given Turing test session probes understanding depends on what the interrogator chooses to ask. The training of ChatGPT does not contain any explicit bias towards demonstrating an understanding of anything, and it seems that a reasonably determined and detached interrogator should have little difficulty in identifying it on this basis.
  - bryan0 4y ago
    I disagree, but I think this is one of the most interesting questions of our time.
- goatlover 4y ago
  Searle didn’t say AGI was impossible, rather the Chinese Room is an argument against GOFAI and symbol manipulation being enough. What makes human brains different is they’re part of an organism interacting with the environment. Symbols get their meaning from use in the environment. The Chinese Room, like chatGPT, is parasitic on those symbols whose meanings we assigned from our use in the real world.
GuB-42 4y ago
To follow up on your mistake, ChatGPT, is, I believe, unable to emulate a Turing machine. A human with a pen and paper can (painstakingly) emulate a Turing machine. In other words ChatGPT completely fails at being a computer! An easy way to trip it is to ask for calculations (123x456=55088?) or to run simple algorithms (ahptblu in reverse is ulbtphah?). Most interestingly, in both these examples, it could write the code that would have given the right answers.
- raydiatian 4y ago
  Related: one class of thought is that the first “AGI” will most likely be the product of another AI, one optimized to design effective per-context models. While ChatGPT itself may not be an effective computer, it almost certainly knows how to “write a program so I can multiply two numbers” and then hand that off to an associated computational machine. The same way that YOU are able to calculated 12757593725274857353527x3747595837353738585837373=…well… https://www.wolframalpha.com/input?i=12757593725274857353527x3747595837353738585837373 https://www.wolframalpha.com/input?i=12757593725274857353527... I don’t think ChatGPT is that “mother” agent that gives birth to the intelligence explosion, but it has demonstrated an exceptional mastery of one of mankind’s greatest tricks: language.
- deleted 4y ago
  [deleted]
- visarga 4y ago
  > “LLMs can’t even do addition” > We show that they CAN add! To teach algos to LLMs, the trick is to describe the algo in enough detail so that there is no room for misinterpretation Algorithmic prompting paper https://arxiv.org/pdf/2211.09066.pdf https://arxiv.org/pdf/2211.09066.pdf
  - rpearl 4y ago
    > the trick is to describe the algo in enough detail so that there is no room for misinterpretation We could create a syntactically unambiguous form of textual representation of instructions. We could call such a piece of text something specific, like a "program".
dragonwriter 4y ago
> ChatGPT seems like it is "just" a very advanced Chinese Room(?). The basic definition of a Chinese Room is that the output is indistinguishable from a human; the whole point of the thought experiment is to illustrate a claim that external behavior cannot be taken as proof of actual understanding. A very advanced Chinese Room would be a system without general understanding that nevertheless behaved as if it were a generally superhuman intelligence.
- cultureswitch 4y ago
  If the Chinese room system is indistinguishable from a "real human being" then they are the same thing.
  - dragonwriter 4y ago
    Well, yes, to a physicalist. The Chinese Room thought experiment is, ultimately, an appeal to dualism.
russdill 4y ago
And who's to say humans are super intelligences and not just a super mockingbird?
- visarga 4y ago
  99.99% of human thought is repetitive, derivative and generally just cached computation being reused. Adding one original thought on top of everything can be a life-long endeavour and often require a PhD. AlphaGo started from scratch and surpassed human level by having access to more experience. It was enough for a neural net to learn this game, because a simulated Go board is close enough to a real one. Remember move 37? More recently AlphaTensor showed a better way to do matrix multiplication than humans could manually discover, also based on massive search followed by verification and learning. Humans appear more intelligent because we have access to validation in a way the AI models don't, we have access to the real world, tools and labs and human society, not just a text dataset or an impoverished simulation. Even so, it's not easy to validate abductive thought. Saying is cheap, proving is what matters. Same for language models - unvalidated generative language models are worthless. Validation is the key. When validating is cheap, a model can beat humans, the neural net architecture is not an obstacle to surpassing humans. When validation is expensive, even humans fumble around - remember how many cooks were around the pot at the CERN particle accelerator a few years ago? All of them sucking on the verification soup. With so many PhD brains, verification was still the scarce ingredient. Without our labs, toys and human peers we can't do it either. One other thing we can't do, for example, is to discover how to build better AI. We just try various ideas out, seeing what sticks. Why can't we just spit out the best idea if we are "intelligent"? Why are we calling working with neural nets a kind of alchemy? Because we haven't verified most of our ideas yet.
  - deleted 4y ago
    [deleted]
- goatlover 4y ago
  A mockingbird of what? We invented language as a communication and survival tool. What would we be mocking? The world?
8n4vidtmkvmk 4y ago
as long as it mocks correct answers, it's fine. but it can't tell which ones are correct and straight up lies and contradicts itself. i guess that makes it more of a con artist
usr1106 4y ago
Inspired by these discussions I wanted to play with ChatGPT. First Coudflare tells me it needs to check "whether I am human", sounds suspiciously like a Turing test :) (I know they don't even try to perform one.) Ironically on this mobile phone their test fails and they don't let me in even after a dozen attempts. (The hardware was made for the Indian market many years ago, so it is not very performant by today's measures. The software is a rather old Firefox, with Javascript, cookies, storage and everything. No idea how they come to the conclusion I would be a bot.)
- runnerup 4y ago
  The old Firefox is a pretty big signal for anti-bot filters. They can generally figure out what hardware you’re on via canvas fingerprinting. Also IPv6 is generally downrated as well. Essentially, if your fingerprint looks different from what they’ve explicitly whitelisted, it’s a strong chance they’ll think you’re a bot.
  - usr1106 4y ago
    Downrating IPv6? I thought most IPv6 users are mobile and phones don't look like an ideal bot platform to me. Either way I use this phone via WLAN and my obsolete ADSL, so definitely no IPv6, the provider has not invested anything for years and announced shutdown several times.

satvikpendem 4y ago

Sounds like you're thinking of a Chinese Room. https://en.wikipedia.org/wiki/Chinese_room https://en.wikipedia.org/wiki/Chinese_room

xyzelement 4y ago

Sorry to be the pedant (someone is gonna do it eventually) - a Turing Machine isn't a machine designed to pass the Turing test (what you are actually referring to)

runnerup 4y ago
Thank you. I have absolutely no idea how I mixed these up despite being very familiar with all the associated concepts and their respective labels.
- amelius 4y ago
  Sounds like a mistake GPT could make, suspicious :)

jannes 4y ago

I think you made a mistake in your first sentence. A turing machine is something different: https://en.wikipedia.org/wiki/Turing_machine https://en.wikipedia.org/wiki/Turing_machine

bryan0 4y ago

I think you’re also misunderstanding the point of the Chinese Room. The Chinese room purports to show that strong AI is not possible because all computer programs cannot “understand”. I think this argument falls apart unless you can explain why our brains are not computable.

runnerup 4y ago
Fair enough. I hope I was clear that I do think that computer programs can understand. I just don't think that optimizing for solving the Turing test will generally yield a computer program capable of "understanding" or "super intelligence".
- chii 4y ago
  A turing test is meant to demonstrate one aspect of understanding/thinking. I mean, what else is a good criteria for understanding?
  - mannykannot 4y ago
    Whether, and to what extent, a given Turing test session probes understanding depends on what the interrogator chooses to ask. The training of ChatGPT does not contain any explicit bias towards demonstrating an understanding of anything, and it seems that a reasonably determined and detached interrogator should have little difficulty in identifying it on this basis.
- bryan0 4y ago
  I disagree, but I think this is one of the most interesting questions of our time.
goatlover 4y ago
Searle didn’t say AGI was impossible, rather the Chinese Room is an argument against GOFAI and symbol manipulation being enough. What makes human brains different is they’re part of an organism interacting with the environment. Symbols get their meaning from use in the environment. The Chinese Room, like chatGPT, is parasitic on those symbols whose meanings we assigned from our use in the real world.

GuB-42 4y ago

To follow up on your mistake, ChatGPT, is, I believe, unable to emulate a Turing machine. A human with a pen and paper can (painstakingly) emulate a Turing machine. In other words ChatGPT completely fails at being a computer! An easy way to trip it is to ask for calculations (123x456=55088?) or to run simple algorithms (ahptblu in reverse is ulbtphah?). Most interestingly, in both these examples, it could write the code that would have given the right answers.

raydiatian 4y ago
Related: one class of thought is that the first “AGI” will most likely be the product of another AI, one optimized to design effective per-context models. While ChatGPT itself may not be an effective computer, it almost certainly knows how to “write a program so I can multiply two numbers” and then hand that off to an associated computational machine. The same way that YOU are able to calculated 12757593725274857353527x3747595837353738585837373=…well… https://www.wolframalpha.com/input?i=12757593725274857353527x3747595837353738585837373 https://www.wolframalpha.com/input?i=12757593725274857353527... I don’t think ChatGPT is that “mother” agent that gives birth to the intelligence explosion, but it has demonstrated an exceptional mastery of one of mankind’s greatest tricks: language.
deleted 4y ago
[deleted]
visarga 4y ago
> “LLMs can’t even do addition” > We show that they CAN add! To teach algos to LLMs, the trick is to describe the algo in enough detail so that there is no room for misinterpretation Algorithmic prompting paper https://arxiv.org/pdf/2211.09066.pdf https://arxiv.org/pdf/2211.09066.pdf
- rpearl 4y ago
  > the trick is to describe the algo in enough detail so that there is no room for misinterpretation We could create a syntactically unambiguous form of textual representation of instructions. We could call such a piece of text something specific, like a "program".

dragonwriter 4y ago

> ChatGPT seems like it is "just" a very advanced Chinese Room(?). The basic definition of a Chinese Room is that the output is indistinguishable from a human; the whole point of the thought experiment is to illustrate a claim that external behavior cannot be taken as proof of actual understanding. A very advanced Chinese Room would be a system without general understanding that nevertheless behaved as if it were a generally superhuman intelligence.

cultureswitch 4y ago
If the Chinese room system is indistinguishable from a "real human being" then they are the same thing.
- dragonwriter 4y ago
  Well, yes, to a physicalist. The Chinese Room thought experiment is, ultimately, an appeal to dualism.

russdill 4y ago

And who's to say humans are super intelligences and not just a super mockingbird?

visarga 4y ago
99.99% of human thought is repetitive, derivative and generally just cached computation being reused. Adding one original thought on top of everything can be a life-long endeavour and often require a PhD. AlphaGo started from scratch and surpassed human level by having access to more experience. It was enough for a neural net to learn this game, because a simulated Go board is close enough to a real one. Remember move 37? More recently AlphaTensor showed a better way to do matrix multiplication than humans could manually discover, also based on massive search followed by verification and learning. Humans appear more intelligent because we have access to validation in a way the AI models don't, we have access to the real world, tools and labs and human society, not just a text dataset or an impoverished simulation. Even so, it's not easy to validate abductive thought. Saying is cheap, proving is what matters. Same for language models - unvalidated generative language models are worthless. Validation is the key. When validating is cheap, a model can beat humans, the neural net architecture is not an obstacle to surpassing humans. When validation is expensive, even humans fumble around - remember how many cooks were around the pot at the CERN particle accelerator a few years ago? All of them sucking on the verification soup. With so many PhD brains, verification was still the scarce ingredient. Without our labs, toys and human peers we can't do it either. One other thing we can't do, for example, is to discover how to build better AI. We just try various ideas out, seeing what sticks. Why can't we just spit out the best idea if we are "intelligent"? Why are we calling working with neural nets a kind of alchemy? Because we haven't verified most of our ideas yet.
- deleted 4y ago
  [deleted]
goatlover 4y ago
A mockingbird of what? We invented language as a communication and survival tool. What would we be mocking? The world?

8n4vidtmkvmk 4y ago

as long as it mocks correct answers, it's fine. but it can't tell which ones are correct and straight up lies and contradicts itself. i guess that makes it more of a con artist

usr1106 4y ago

Inspired by these discussions I wanted to play with ChatGPT. First Coudflare tells me it needs to check "whether I am human", sounds suspiciously like a Turing test :) (I know they don't even try to perform one.) Ironically on this mobile phone their test fails and they don't let me in even after a dozen attempts. (The hardware was made for the Indian market many years ago, so it is not very performant by today's measures. The software is a rather old Firefox, with Javascript, cookies, storage and everything. No idea how they come to the conclusion I would be a bot.)

runnerup 4y ago
The old Firefox is a pretty big signal for anti-bot filters. They can generally figure out what hardware you’re on via canvas fingerprinting. Also IPv6 is generally downrated as well. Essentially, if your fingerprint looks different from what they’ve explicitly whitelisted, it’s a strong chance they’ll think you’re a bot.
- usr1106 4y ago
  Downrating IPv6? I thought most IPv6 users are mobile and phones don't look like an ideal bot platform to me. Either way I use this phone via WLAN and my obsolete ADSL, so definitely no IPv6, the provider has not invested anything for years and announced shutdown several times.