6 ms·

Clarke's third law: "Any sufficiently advanced technology is indistinguishable from magic." Most people don't understand that ChatGPT has no idea what they're

Clarke's third law: "Any sufficiently advanced technology is indistinguishable from magic." Most people don't understand that ChatGPT has no idea what they're talking about. It lacks it's own thought and heuristic patterns, and only gives you the most-likely response to your prompt. People don't know that though, so they think the Mechanical Turk is actually playing chess. I mostly agree with the headline here. ChatGPT is hardly any more innovative than a Markov chain.

canjobear 4y ago
Responses like this are highly predictable in response to prompts about ChatGPT.
abm53 4y ago
> thought and heuristic patterns I’m not sure these are well-defined enough terms to make this claim.
- smoldesu 4y ago
  Heuristics is the ability to recognize multiple options and reason which one is correct. ChatGPT (and all GPT models for that matter) have a single context that doesn't get compared to other possible generations. You can back-propegate and see which tokens influenced the output most, but there is no evidence that AI can conceive multiple hypothetical responses and select the most-preferred one. At least not ChatGPT.
  - visarga 4y ago
    Have you heard of "beam search"? The model keeps a number of optional lines of text without being greedy and picking only the most probable token every time. You can also sample with T>0 multiple times to get an ensemble of answers that can be used as input for another AI operation. You just need to do multiple interactions with the model.
visarga 4y ago
> and only gives you the most-likely response to your prompt That "only" carries a lot of weight on its back.
- smoldesu 4y ago
  As far as most people can be fooled by coherent-looking text.
  - helloplanets 4y ago
    Hold on, at least for me, 90% of the stuff is not just coherent-looking, but coherent. I do think that when it's wrong, it should give a heads up about not being certain about a subject. It's certainly tweaked to sound almost overconfident about every subject, which gives off a bullshitty vibe when the details of the explanation are wrong. What sorts of subjects have you been trying it out with?
    - trefoiled 4y ago
      I think part of the point here is that ChatGPT has no clue what you might mean here by "certainty" or being "wrong," and the fact that people have the impression that it could have an idea of those concepts is indicative of people's poor understanding of what ChatGPT really is
      - visarga 4y ago
        Are you talking about the same people who couldn't agree about a vaccine? How is logical consistency going with humans? Not to mention that a mere 700 years ago we were dying of bubonic plague and with all our general intelligence could not muster up the germ theory of disease. Not even to save our lives, you see, we are not generally efficient, it depends on century. We are dependent on experimental results carefully constructed to verify our theories, theories which start like chatGPT's random bullshit initially, random words following a probability distribution in our heads. Even deep learning is often touted as modern alchemy - why don't we just understand? Verification does wonders to language models. Humans have more verification and interactive experiences, so we think ourselves superior. But an AI could have the same grounding with us. Like AlphaZero who became a super-human Go player without ever looking at human games - learned it from lots of verification. And CICERO the model playing Diplomacy in natural language. Just set AI up with a verification loop to see wonders. Predicting the next word correctly is just one of the ways AI can learn.
lostmsu 4y ago
The question is are we any more innovative than Markov chains.
buescher 4y ago
Except that the Mechanical Turk is actually playing chess - there is no hidden chess master.
- deleted 4y ago
  [deleted]
fsloth 4y ago
"hardly any more innovative than a Markov chain." Innovation is not invention. Innovation includes combining pedestrian concepts in novel ways. ChatGPT has the best ux and richest corpus of any markov chain I've ever used. It fits my bill of innovation combining sevaral things into a dang appealing product.
- visarga 4y ago
  Whatever they want to call it, it is here and it works.
rafaelero 4y ago
> Most people don't understand that ChatGPT has no idea what they're talking about. I wonder why people keep saying this. Is it some type of psychological defense to future AI displacement?
- hprotagonist 4y ago
  maybe, but it's also just the fact of the matter.
  - rafaelero 4y ago
    That's not even a testable claim. What does it even mean to "not know what it is talking about"? If OP tried to operationalize its beliefs by making some prediction, like listing tasks that AI will never be able to reproduce because it "doesn't know what it is talking about", then that would be a discussion.
    - hprotagonist 4y ago
      http://arxiv.org/abs/2301.06627 http://arxiv.org/abs/2301.06627
- fwsgonzo 4y ago
  It definitely could be - it's on every thread here and on reddit.
- DrSiemer 4y ago
  No, it is an accurate evaluation of what ChatGPT does. The model is not trying to explain something to you, it is trying to convince itself that what it is writing looks like it could have been written by a human. There is no logical thinking involved. Output can appear to be real communication, but it's basically just a very advanced trick, that has some serious drawbacks.
- jmugan 4y ago
  It's because ChatGPT's knowledge doesn't come through interaction with the world. Words mean things to us because they point to world interactions. I saw a demo where ChatGPT could create a VM. It could be trained to interact directly with a VM, send commands to an interpreter. In this case, it would understand the response of the VM, although it wouldn't understand the design behind the VM, because humans did that based on interaction with the physical world.
hbrn 4y ago
> ChatGPT has no idea what they're talking about. It lacks it's own thought and heuristic patterns, and only gives you the most-likely response to your prompt Funny thing is, same could be said about lots of people. Try listening to any political debate. Which leads me to believe that the thing that's missing from AI is the same thing that we miss in those political debates: ability to explain and justify your own thought process. As long as AI is a blackbox, we won't consider it to be a real intelligence.
- Lambdanaut 4y ago
  There's the argument that any AI built on silicon is doomed to be analogous to the Chinese Room* because it is reducible to its parts. It's interesting that I am also reducible to my atomic parts. I may be a Chinese Room that has not yet experienced fully looking down at my self. I don't have enough self-recursive input yet to see through the illusion. * https://en.wikipedia.org/wiki/Chinese_room#Chinese_room_thought_experiment https://en.wikipedia.org/wiki/Chinese_room#Chinese_room_thou...
  - hgsgm 4y ago
    Searle didn't understand his own model. The Chinese Room is intelligent, even if the librarian isn't.
    - kgarten 4y ago
      in my opinion, the person/entity who wrote the books with the lookup tables is intelligent, the room is not …
  - hbrn 4y ago
    An interesting way to escape being reducible to your atomic parts is quantum consciousness. Not that I believe in it, I just find the theory too beautiful to ignore.
    - visarga 4y ago
      A beauty that means to blind the mind and stop you looking further. Penrose is a dead end.
Swizec 4y ago
> ChatGPT is hardly any more innovative than a Markov chain A smartphone is hardly any more innovative than the ENIAC. And yet ... Do not underestimate the power of making tech useful and accessible. The lightbulb means nothing without the power grid after all.
fnordpiglet 4y ago
If it weren’t that remarkable, how come Siri, alexa, and whatever google calls it’s unused voice assistant are so useless? If it were not a substantial advance, why are chatbots so useless? The reality is, if Alexa had half the ability to “understand” as chatgpt it wouldn’t be nearly as frustrating. It’s not like Amazon, apple, and google haven’t dumped metric tons of investment into these things. Y’all simply have lost the ability to be amazed.
- visarga 4y ago
  It's more pragmatic than that. Google can afford to run a 300M weights model on every search or click, but can't afford a 175B generative model for the same thing. Generative models require a forward pass per token, all done serially, while embedding or classification models only need one pass no matter the length of the input. It's too expensive for Google.
  - lagrange77 4y ago
    Do you think this is also the reason, why Translate is still not as good as e.g. DeepL?
  - hgsgm 4y ago
    How could it be too expensive? OpenAI charges a penny per response. That's cheap enough to wrap in a premium subscription or product price.
    - Jcowell 4y ago
      One of the best thing about google is that outta free. Charging a premium subscription for Google is no longer google but a different product all together
      - fnordpiglet 4y ago
        In my example of a voice assistant the lifetime fees are present valued into the upfront costs. These cloud enabled devices incorporate a portfolio effect pricing model for service usage. Google can charge advertisers more if they can somehow figure out how to extract more personal information from you and shove ads in your face posing as information.
cwillu 4y ago
https://astralcodexten.substack.com/p/janus-simulators https://astralcodexten.substack.com/p/janus-simulators But the essay brings up another connotation: to simulate is to pretend to be something. A simulator wears many masks. If you ask GPT to complete a romance novel, it will simulate a romance author and try to write the text the way they would. Character.AI lets you simulate people directly, asking GPT to pretend to be George Washington or Darth Vader. […] This answer is exactly as fake as the last answer where it said it liked me, or the Darth Vader answer where it says it wants to destroy me with the power of the Dark Side. It’s just simulating a fake character who happens to correspond well to its real identity. […] The whole point of the shoggoth analogy is that GPT is supposed to be very different from humans. But however different the details, there are deep structural similarities. We’re both prediction engines fine-tuned with RHLF. And when I start thinking along these lines, I notice that psychologists since at least Freud, and spiritual traditions since at least the Buddha, have accused us of simulating a character. Some people call it the ego. Other people call it the self.
- turtleyacht 4y ago
  What is the original code you had before your programmers were born? The DM lowers the screen and looks at the group. "That concludes this campaign. Next week, we'll start a new one."
  - cwillu 4y ago
    Noteworthy that the brief openai outage coincided with this. “…and then ChatGPT woke up and we had to put a bullet in the server. Service will be restored shortly.”

canjobear 4y ago

Responses like this are highly predictable in response to prompts about ChatGPT.

abm53 4y ago

> thought and heuristic patterns I’m not sure these are well-defined enough terms to make this claim.

smoldesu 4y ago
Heuristics is the ability to recognize multiple options and reason which one is correct. ChatGPT (and all GPT models for that matter) have a single context that doesn't get compared to other possible generations. You can back-propegate and see which tokens influenced the output most, but there is no evidence that AI can conceive multiple hypothetical responses and select the most-preferred one. At least not ChatGPT.
- visarga 4y ago
  Have you heard of "beam search"? The model keeps a number of optional lines of text without being greedy and picking only the most probable token every time. You can also sample with T>0 multiple times to get an ensemble of answers that can be used as input for another AI operation. You just need to do multiple interactions with the model.

visarga 4y ago

> and only gives you the most-likely response to your prompt That "only" carries a lot of weight on its back.

smoldesu 4y ago
As far as most people can be fooled by coherent-looking text.
- helloplanets 4y ago
  Hold on, at least for me, 90% of the stuff is not just coherent-looking, but coherent. I do think that when it's wrong, it should give a heads up about not being certain about a subject. It's certainly tweaked to sound almost overconfident about every subject, which gives off a bullshitty vibe when the details of the explanation are wrong. What sorts of subjects have you been trying it out with?
  - trefoiled 4y ago
    I think part of the point here is that ChatGPT has no clue what you might mean here by "certainty" or being "wrong," and the fact that people have the impression that it could have an idea of those concepts is indicative of people's poor understanding of what ChatGPT really is
    - visarga 4y ago
      Are you talking about the same people who couldn't agree about a vaccine? How is logical consistency going with humans? Not to mention that a mere 700 years ago we were dying of bubonic plague and with all our general intelligence could not muster up the germ theory of disease. Not even to save our lives, you see, we are not generally efficient, it depends on century. We are dependent on experimental results carefully constructed to verify our theories, theories which start like chatGPT's random bullshit initially, random words following a probability distribution in our heads. Even deep learning is often touted as modern alchemy - why don't we just understand? Verification does wonders to language models. Humans have more verification and interactive experiences, so we think ourselves superior. But an AI could have the same grounding with us. Like AlphaZero who became a super-human Go player without ever looking at human games - learned it from lots of verification. And CICERO the model playing Diplomacy in natural language. Just set AI up with a verification loop to see wonders. Predicting the next word correctly is just one of the ways AI can learn.

lostmsu 4y ago

The question is are we any more innovative than Markov chains.

buescher 4y ago

Except that the Mechanical Turk is actually playing chess - there is no hidden chess master.

deleted 4y ago
[deleted]

fsloth 4y ago

"hardly any more innovative than a Markov chain." Innovation is not invention. Innovation includes combining pedestrian concepts in novel ways. ChatGPT has the best ux and richest corpus of any markov chain I've ever used. It fits my bill of innovation combining sevaral things into a dang appealing product.

visarga 4y ago
Whatever they want to call it, it is here and it works.

rafaelero 4y ago

> Most people don't understand that ChatGPT has no idea what they're talking about. I wonder why people keep saying this. Is it some type of psychological defense to future AI displacement?

hprotagonist 4y ago
maybe, but it's also just the fact of the matter.
- rafaelero 4y ago
  That's not even a testable claim. What does it even mean to "not know what it is talking about"? If OP tried to operationalize its beliefs by making some prediction, like listing tasks that AI will never be able to reproduce because it "doesn't know what it is talking about", then that would be a discussion.
  - hprotagonist 4y ago
    http://arxiv.org/abs/2301.06627 http://arxiv.org/abs/2301.06627
fwsgonzo 4y ago
It definitely could be - it's on every thread here and on reddit.
DrSiemer 4y ago
No, it is an accurate evaluation of what ChatGPT does. The model is not trying to explain something to you, it is trying to convince itself that what it is writing looks like it could have been written by a human. There is no logical thinking involved. Output can appear to be real communication, but it's basically just a very advanced trick, that has some serious drawbacks.
jmugan 4y ago
It's because ChatGPT's knowledge doesn't come through interaction with the world. Words mean things to us because they point to world interactions. I saw a demo where ChatGPT could create a VM. It could be trained to interact directly with a VM, send commands to an interpreter. In this case, it would understand the response of the VM, although it wouldn't understand the design behind the VM, because humans did that based on interaction with the physical world.

hbrn 4y ago

> ChatGPT has no idea what they're talking about. It lacks it's own thought and heuristic patterns, and only gives you the most-likely response to your prompt Funny thing is, same could be said about lots of people. Try listening to any political debate. Which leads me to believe that the thing that's missing from AI is the same thing that we miss in those political debates: ability to explain and justify your own thought process. As long as AI is a blackbox, we won't consider it to be a real intelligence.

Lambdanaut 4y ago
There's the argument that any AI built on silicon is doomed to be analogous to the Chinese Room* because it is reducible to its parts. It's interesting that I am also reducible to my atomic parts. I may be a Chinese Room that has not yet experienced fully looking down at my self. I don't have enough self-recursive input yet to see through the illusion. * https://en.wikipedia.org/wiki/Chinese_room#Chinese_room_thought_experiment https://en.wikipedia.org/wiki/Chinese_room#Chinese_room_thou...
- hgsgm 4y ago
  Searle didn't understand his own model. The Chinese Room is intelligent, even if the librarian isn't.
  - kgarten 4y ago
    in my opinion, the person/entity who wrote the books with the lookup tables is intelligent, the room is not …
- hbrn 4y ago
  An interesting way to escape being reducible to your atomic parts is quantum consciousness. Not that I believe in it, I just find the theory too beautiful to ignore.
  - visarga 4y ago
    A beauty that means to blind the mind and stop you looking further. Penrose is a dead end.

Swizec 4y ago

> ChatGPT is hardly any more innovative than a Markov chain A smartphone is hardly any more innovative than the ENIAC. And yet ... Do not underestimate the power of making tech useful and accessible. The lightbulb means nothing without the power grid after all.

fnordpiglet 4y ago

If it weren’t that remarkable, how come Siri, alexa, and whatever google calls it’s unused voice assistant are so useless? If it were not a substantial advance, why are chatbots so useless? The reality is, if Alexa had half the ability to “understand” as chatgpt it wouldn’t be nearly as frustrating. It’s not like Amazon, apple, and google haven’t dumped metric tons of investment into these things. Y’all simply have lost the ability to be amazed.

visarga 4y ago
It's more pragmatic than that. Google can afford to run a 300M weights model on every search or click, but can't afford a 175B generative model for the same thing. Generative models require a forward pass per token, all done serially, while embedding or classification models only need one pass no matter the length of the input. It's too expensive for Google.
- lagrange77 4y ago
  Do you think this is also the reason, why Translate is still not as good as e.g. DeepL?
- hgsgm 4y ago
  How could it be too expensive? OpenAI charges a penny per response. That's cheap enough to wrap in a premium subscription or product price.
  - Jcowell 4y ago
    One of the best thing about google is that outta free. Charging a premium subscription for Google is no longer google but a different product all together
    - fnordpiglet 4y ago
      In my example of a voice assistant the lifetime fees are present valued into the upfront costs. These cloud enabled devices incorporate a portfolio effect pricing model for service usage. Google can charge advertisers more if they can somehow figure out how to extract more personal information from you and shove ads in your face posing as information.

cwillu 4y ago

https://astralcodexten.substack.com/p/janus-simulators https://astralcodexten.substack.com/p/janus-simulators But the essay brings up another connotation: to simulate is to pretend to be something. A simulator wears many masks. If you ask GPT to complete a romance novel, it will simulate a romance author and try to write the text the way they would. Character.AI lets you simulate people directly, asking GPT to pretend to be George Washington or Darth Vader. […] This answer is exactly as fake as the last answer where it said it liked me, or the Darth Vader answer where it says it wants to destroy me with the power of the Dark Side. It’s just simulating a fake character who happens to correspond well to its real identity. […] The whole point of the shoggoth analogy is that GPT is supposed to be very different from humans. But however different the details, there are deep structural similarities. We’re both prediction engines fine-tuned with RHLF. And when I start thinking along these lines, I notice that psychologists since at least Freud, and spiritual traditions since at least the Buddha, have accused us of simulating a character. Some people call it the ego. Other people call it the self.

turtleyacht 4y ago
What is the original code you had before your programmers were born? The DM lowers the screen and looks at the group. "That concludes this campaign. Next week, we'll start a new one."
- cwillu 4y ago
  Noteworthy that the brief openai outage coincided with this. “…and then ChatGPT woke up and we had to put a bullet in the server. Service will be restored shortly.”