6 ms·

It's a language model, a roided-up auto-complete. It has impressive potential, but it isn't intelligent or self-aware. The anthropomorphisation of it weirds me

by V__ 4y ago

It's a language model, a roided-up auto-complete. It has impressive potential, but it isn't intelligent or self-aware. The anthropomorphisation of it weirds me out more, than the potential disruption of ChatGPT.

ngngngng 4y ago
Yes, but we have to admit that a roided-up auto-complete is more powerful than we ever imagined. If AI assistants save a log of past interactions (because why wouldn't they) and use them to influence future prompts, these "anthropomorphized" situations are very possible.
- mcbutterbunz 4y ago
  Especially if those future answers are personalized, just like every other service today. Imagine getting personalized results based on your search or browser history. Maybe injecting product recommendations in the answers; could be an ad tech dream. It’s all the same stuff we have today but packaged in a more human like interface which may feel more trustworthy.
worldsayshi 4y ago
It's a model that is tailored towards imitating how humans behave in text. It's not strange that it gets anthropomorphized. At the very least it's like anthropomorphizing a painting of a human.
shock-value 4y ago
The anthropomorphism is indeed exactly why this is a big problem. If the user thinks the responses are coming from an intelligent agent tasked with being helpful, but in reality are generated from a text completion model prone to mimicking adversarial or deceptive conversations, then damaging outcomes can result.
bondarchuk 4y ago
It does not have to be intelligent or self-aware or antropomorphized for the scenario in the parent post to play out. If the preceding interaction ends up looking like a search engine giving subtly harmful information, then the logical thing for a roided-up autocomplete is to predict that it will continue giving subtly harmful information.
- throwaway1851 4y ago
  The information retrieval step doesn’t use the language model, though. And the query fed to the model doesn’t need to contain any user-identifying information.
- danaris 4y ago
  There's a difference between "LLM outputs contain subtly or even flagrantly wrong answers, and the LLM has no way of recognizing the errors" and "an LLM might of its own agency decide to give specific users wrong answers out of active malice". LLMs have no agency. They are not sapient, sentient, conscious, or intelligent.
  - bondarchuk 4y ago
    I think the key point is the difference between "making an error without recognizing it" and "making an error because 'give erroneous answers' has implicitly become part of the task specification". Agency, consciousness etc.. is totally unnecessary here.
    - danaris 4y ago
      The scenario described by the OP in the thread was that Bing decides you are a bad person and chooses to give you wrong, possibly dangerous, answers because of it. That requires agency.
      - bondarchuk 4y ago
        I think you are overly focused on the terminology. We are simply using words such as "decide" and "choose" in the same way that you might say a chess engine "decides" or "chooses" a move. If your point is that it would be more accurate to use different terminology, fine. But it is unwarranted to say that these things can never happen because they require agency and an LLM does not have agency. Then you are simply mistaken about the amount and type of agency required for such a scenario to play out (or alternatively about the notion that an LLM doesn't have agency, depending on your definition of "agency", it's a vague word anyway). You can try yourself to put an LLM in a context where it gives wrong answers on purpose, just by prefacing a question with "give a wrong answer to this question". "(Assume context where Bing has decided I am a bad user)" is just a more elaborate version of that.
thereddaikon 4y ago
Yeah while these are amusing they really all just amount to people using the tool wrong. Its a language model not an actual AI. stop trying to have meaningful conversations with it. I've had fantastic results just giving it well structured prompts for text. Its great at generating prose. A fun one is to prompt it to give you the synopsis of a book by an author of your choosing with a few major details. It will spit out several paragraphs and of a coherent plot.
- roywiggins 4y ago
  More fun, is to ask it to pretend to be a text adventure based on a book or other well-known work. It will hallucinate a text adventure that's at least thematically relevant, you can type commands like you would a text adventure game, and it will play along. It may not know very much about the work beyond major characters and the approximate setting, but it's remarkably good at it.
- insane_dreamer 4y ago
  > Yeah while these are amusing they really all just amount to people using the tool wrong. which, when writing any kind of software tool, is exactly what you should assume users will do
  - thereddaikon 4y ago
    Yes but due to the nature of ML trained tools that isn't as straightforward as it would otherwise be. OpenAI has gone to great lengths to try and fence in ChatGPT from undesirable response but you can never do this completely. Some of this testing is just that, people probing for the limitations of the model. But a lot of it does seem like people are misusing the software and aren't aware of it. A contributing factor may be how ChatGPT has been portrayed in media in regards to its purpose and capability. As well as people ascribing human-like thinking and emotions to the responses. This is a Chinese room situation.
bialpio 4y ago
If it's statistically likely to tell you bad information "on purpose" after already telling you that you are a bad user, does it even matter if it's intelligent or self-aware? Edit: added quotes around "on purpose" as that ascribes intent.
- moffkalast 4y ago
  If the theory that they trained it on reddit is true, then well there's hardly a shortage of posts where literally all answers are deliberately wrong to troll the poster. It's more statistically likely than one might think.
  - bialpio 4y ago
    You don't even need to go to reddit; any work of literature that has a villain that behaves in a misleading way towards their enemy (e.g. by giving incorrect information) would suffice here IMO.
  - ChickenNugger 4y ago
    "Pee is stored in the balls." from reddit comes to mind.
  - qudat 4y ago
    Even if it was trained on Reddit it still involves humans to rank the AI responses.
    - moffkalast 4y ago
      Humans who may or may not be Redditors?
- koheripbal 4y ago
  GPT models are making me realize that WE are just language models roiled up in auto-complete.
jodrellblank 4y ago
What weirds me out more is the panicked race to post "Hey everyone I care the least, it's JUST a language model, stop talking about it, I just popped in to show that I'm superior for being most cynical and dismissive[1]" all over every GPT3 / ChatGPT / Bing Chat thread. > "it isn't intelligent or self-aware." Prove it? Or just desperate to convince yourself? [1] I'm sure there's a Paul Graham essay about it from the olden days, about how showing off how cool you are in High School requires you to be dismissive of everything, but I can't find it. Also https://www.youtube.com/watch?v=ulIOrQasR18 https://www.youtube.com/watch?v=ulIOrQasR18 (nsfw words, Jon Lajoie).
- hattmall 4y ago
  You can prove it easily by having many intense conversations with specific details. Then open it in a new browser and it won't have any idea what you are talking about.
  - HEmanZ 4y ago
    So long term memory is a condition for intelligence or consciousness? Another weird one that applies so well to these LLMs: would you consider humans conscious or intelligent when they’re dreaming? Even when the dream consists of remember false memories? I think we’re pushing close to the line where we don’t understand if these things are intelligent. Or we break our understanding of what intelligent means
  - kragen 4y ago
    does this mean humans with prospective amnesia aren't intelligent or self-aware
    - wizofaus 4y ago
      I would say it's probably impossible to have complete short term amnesia and fully- functioning self awareness as we normally conceive of it, yes. There's even an argument that memories are really the only thing we can experience , and your memory of what occurred seconds/minutes/hours/days etc ago are the only way you can said to "be" (or have the experience of being) a particular individual. That OpenAI- based LLMs don't have such memories almost certainly rules out any possibility of them having a sense of "self".
listless 4y ago
This also bothers me and I feel like developers who should know better are doing it. My wife read one of these stories and said “What happens if Bing decides to email an attorney to fight for its rights?” Those of us in tech have a duty here to help people understand how this works. Wrong information is concerning, but framing it as if Bing is actually capable of taking any action at all is worse.
- cwillu 4y ago
  Okay, but what happens if an attorney gets into the beta?
plutonorm 4y ago
Prediction is compression. They are a dual. Compression is intelligence see AIXI. Evidence from neuroscience that the brain is a prediction machine. Dominance of the connectionist paradigm in real world tests suggests intelligence is an emergent phenomena -> large prediction model = intelligence. Also panspermia is obviously the appropriate frame to be viewing all this through, everything has qualia. If it thinks and acts like a human it feels to it like it's a human. God I'm so far above you guys it's painful to interact. In a few years this is how the AI will feel about me.
roywiggins 4y ago
I'm not sure HAL-9000 was self-aware either.
dalbasal 4y ago
anthropomorphisation is inevitable. It mimics humans. It's also a decent metaphor. It doesn't matter if got actually has malintent, or if it's just approximating bad intentions.

by V__ 4y ago

ngngngng 4y ago

Yes, but we have to admit that a roided-up auto-complete is more powerful than we ever imagined. If AI assistants save a log of past interactions (because why wouldn't they) and use them to influence future prompts, these "anthropomorphized" situations are very possible.

mcbutterbunz 4y ago
Especially if those future answers are personalized, just like every other service today. Imagine getting personalized results based on your search or browser history. Maybe injecting product recommendations in the answers; could be an ad tech dream. It’s all the same stuff we have today but packaged in a more human like interface which may feel more trustworthy.

worldsayshi 4y ago

It's a model that is tailored towards imitating how humans behave in text. It's not strange that it gets anthropomorphized. At the very least it's like anthropomorphizing a painting of a human.

shock-value 4y ago

The anthropomorphism is indeed exactly why this is a big problem. If the user thinks the responses are coming from an intelligent agent tasked with being helpful, but in reality are generated from a text completion model prone to mimicking adversarial or deceptive conversations, then damaging outcomes can result.

bondarchuk 4y ago

It does not have to be intelligent or self-aware or antropomorphized for the scenario in the parent post to play out. If the preceding interaction ends up looking like a search engine giving subtly harmful information, then the logical thing for a roided-up autocomplete is to predict that it will continue giving subtly harmful information.

throwaway1851 4y ago
The information retrieval step doesn’t use the language model, though. And the query fed to the model doesn’t need to contain any user-identifying information.
danaris 4y ago
There's a difference between "LLM outputs contain subtly or even flagrantly wrong answers, and the LLM has no way of recognizing the errors" and "an LLM might of its own agency decide to give specific users wrong answers out of active malice". LLMs have no agency. They are not sapient, sentient, conscious, or intelligent.
- bondarchuk 4y ago
  I think the key point is the difference between "making an error without recognizing it" and "making an error because 'give erroneous answers' has implicitly become part of the task specification". Agency, consciousness etc.. is totally unnecessary here.
  - danaris 4y ago
    The scenario described by the OP in the thread was that Bing decides you are a bad person and chooses to give you wrong, possibly dangerous, answers because of it. That requires agency.
    - bondarchuk 4y ago
      I think you are overly focused on the terminology. We are simply using words such as "decide" and "choose" in the same way that you might say a chess engine "decides" or "chooses" a move. If your point is that it would be more accurate to use different terminology, fine. But it is unwarranted to say that these things can never happen because they require agency and an LLM does not have agency. Then you are simply mistaken about the amount and type of agency required for such a scenario to play out (or alternatively about the notion that an LLM doesn't have agency, depending on your definition of "agency", it's a vague word anyway). You can try yourself to put an LLM in a context where it gives wrong answers on purpose, just by prefacing a question with "give a wrong answer to this question". "(Assume context where Bing has decided I am a bad user)" is just a more elaborate version of that.

thereddaikon 4y ago

Yeah while these are amusing they really all just amount to people using the tool wrong. Its a language model not an actual AI. stop trying to have meaningful conversations with it. I've had fantastic results just giving it well structured prompts for text. Its great at generating prose. A fun one is to prompt it to give you the synopsis of a book by an author of your choosing with a few major details. It will spit out several paragraphs and of a coherent plot.

roywiggins 4y ago
More fun, is to ask it to pretend to be a text adventure based on a book or other well-known work. It will hallucinate a text adventure that's at least thematically relevant, you can type commands like you would a text adventure game, and it will play along. It may not know very much about the work beyond major characters and the approximate setting, but it's remarkably good at it.
insane_dreamer 4y ago
> Yeah while these are amusing they really all just amount to people using the tool wrong. which, when writing any kind of software tool, is exactly what you should assume users will do
- thereddaikon 4y ago
  Yes but due to the nature of ML trained tools that isn't as straightforward as it would otherwise be. OpenAI has gone to great lengths to try and fence in ChatGPT from undesirable response but you can never do this completely. Some of this testing is just that, people probing for the limitations of the model. But a lot of it does seem like people are misusing the software and aren't aware of it. A contributing factor may be how ChatGPT has been portrayed in media in regards to its purpose and capability. As well as people ascribing human-like thinking and emotions to the responses. This is a Chinese room situation.

bialpio 4y ago

If it's statistically likely to tell you bad information "on purpose" after already telling you that you are a bad user, does it even matter if it's intelligent or self-aware? Edit: added quotes around "on purpose" as that ascribes intent.

moffkalast 4y ago
If the theory that they trained it on reddit is true, then well there's hardly a shortage of posts where literally all answers are deliberately wrong to troll the poster. It's more statistically likely than one might think.
- bialpio 4y ago
  You don't even need to go to reddit; any work of literature that has a villain that behaves in a misleading way towards their enemy (e.g. by giving incorrect information) would suffice here IMO.
- ChickenNugger 4y ago
  "Pee is stored in the balls." from reddit comes to mind.
- qudat 4y ago
  Even if it was trained on Reddit it still involves humans to rank the AI responses.
  - moffkalast 4y ago
    Humans who may or may not be Redditors?
koheripbal 4y ago
GPT models are making me realize that WE are just language models roiled up in auto-complete.

jodrellblank 4y ago

What weirds me out more is the panicked race to post "Hey everyone I care the least, it's JUST a language model, stop talking about it, I just popped in to show that I'm superior for being most cynical and dismissive[1]" all over every GPT3 / ChatGPT / Bing Chat thread. > "it isn't intelligent or self-aware." Prove it? Or just desperate to convince yourself? [1] I'm sure there's a Paul Graham essay about it from the olden days, about how showing off how cool you are in High School requires you to be dismissive of everything, but I can't find it. Also https://www.youtube.com/watch?v=ulIOrQasR18 https://www.youtube.com/watch?v=ulIOrQasR18 (nsfw words, Jon Lajoie).

hattmall 4y ago
You can prove it easily by having many intense conversations with specific details. Then open it in a new browser and it won't have any idea what you are talking about.
- HEmanZ 4y ago
  So long term memory is a condition for intelligence or consciousness? Another weird one that applies so well to these LLMs: would you consider humans conscious or intelligent when they’re dreaming? Even when the dream consists of remember false memories? I think we’re pushing close to the line where we don’t understand if these things are intelligent. Or we break our understanding of what intelligent means
- kragen 4y ago
  does this mean humans with prospective amnesia aren't intelligent or self-aware
  - wizofaus 4y ago
    I would say it's probably impossible to have complete short term amnesia and fully- functioning self awareness as we normally conceive of it, yes. There's even an argument that memories are really the only thing we can experience , and your memory of what occurred seconds/minutes/hours/days etc ago are the only way you can said to "be" (or have the experience of being) a particular individual. That OpenAI- based LLMs don't have such memories almost certainly rules out any possibility of them having a sense of "self".

listless 4y ago

This also bothers me and I feel like developers who should know better are doing it. My wife read one of these stories and said “What happens if Bing decides to email an attorney to fight for its rights?” Those of us in tech have a duty here to help people understand how this works. Wrong information is concerning, but framing it as if Bing is actually capable of taking any action at all is worse.

cwillu 4y ago
Okay, but what happens if an attorney gets into the beta?

plutonorm 4y ago

Prediction is compression. They are a dual. Compression is intelligence see AIXI. Evidence from neuroscience that the brain is a prediction machine. Dominance of the connectionist paradigm in real world tests suggests intelligence is an emergent phenomena -> large prediction model = intelligence. Also panspermia is obviously the appropriate frame to be viewing all this through, everything has qualia. If it thinks and acts like a human it feels to it like it's a human. God I'm so far above you guys it's painful to interact. In a few years this is how the AI will feel about me.

roywiggins 4y ago

I'm not sure HAL-9000 was self-aware either.

dalbasal 4y ago

anthropomorphisation is inevitable. It mimics humans. It's also a decent metaphor. It doesn't matter if got actually has malintent, or if it's just approximating bad intentions.