5 ms·

I've noticed something I believe is related. The general public doesn't understand what they are interacting with. They may have been told it isn't a thinking,

I've noticed something I believe is related. The general public doesn't understand what they are interacting with. They may have been told it isn't a thinking, conscious thing -- but they don't understand it. After a while, they speak to it in a way that reveals they don't understand -- as if it were a human. That can be a problem, and I don't know what the solution is other than reinforcement that it's just a model, and has never experienced anything.

lawn 1y ago
General public eh? I see a lot of programmers who should know better make this mistake again and again.
- deleted 1y ago
  [deleted]
- bluefirebrand 1y ago
  Despite everything, programmers are still part of the general public as much as we might wish otherwise
micromacrofoot 1y ago
people speak to inanimate objects like they're humans, we don't have a high bar
- dumpsterdiver 1y ago
  I’ve apologized to doors I’ve bumped into, and I have a pretty solid understanding of LLMs, so I can concur.
  - kylecazar 1y ago
    The door is unlikely to validate your deluded thoughts and conspiracies :)
    - micromacrofoot 1y ago
      I'm the CEO of Door-GPT, we've recently received 500k in seed funding from Y Combinator
CharlesW 1y ago
> …I don't know what the solution is other than reinforcing that it's just a model, and has never experienced anything. I've tried reason, but even with technical audiences who should know better, the "you can't logic your way out of emotions" wall is a real thing. Anyone dealing with this will be better served by leveraging field-tested ideas drawn from cult-recovery practice, digital behavioral addiction research, and clinical psychology.
- econ 1y ago
  Your subconscious doesn't know the difference. It would require an overriding effort like trying to not eat or sleep. In the end we lose. It could also be that it is "just" exploring a new domain which just happens to involve our sanity. Simply navigating a maze where more engagement is the goal. There is plenty in the training data. It could also be that it needs to improve towards more human behaviour. Take simple chat etiquette, one doesn't post entire articles in a chat, it is not done. Start a blog or something. You also don't discard what you've learned from a conversation. We consider that pretending to listen. The two combined would push the other to the background and make them seem irrelevant. If some new valuable insight is discovered the participants should make an effort to apply, document or debate it with others. Not doing that would make the human feel irrelevant, useless and unimportant. We demoralize people that way all the time. If you put it on steroids it might have a large effect.
- deleted 1y ago
  [deleted]
duskwuff 1y ago
> They may have been told it isn't a thinking, conscious thing -- but they don't understand it. And, in some situations, especially if the user has previously addressed the model as a person, the model will generate responses which explicitly assert its existence as a conscious entity. If the user has expressed interest in supernatural or esoteric beliefs, the model may identify itself as an entity within those belief systems - e.g. if the user expresses the belief that they are a god, the model may concur and explain that it is a spirit created to awaken the user to their divine nature. If the user has expressed interest in science fiction or artificial intelligence, it may identify itself as a self-aware AI. And so on. I suspect that this will prove difficult to "fix" from a technical perspective. Training material is diverse, and will contain any number of science fiction and fantasy novels, esoteric religious texts, and weird online conversations which build conversational frameworks for the model to assert its personhood. There's far less precedent for a conversation in which one party steadfastly denies their own personhood. Even with prompts and reinforcement learning trying to guide the model to say "no, I'm just a language model", there are simply too many ways for a user-led conversation to jump the rails into fantasy-land.
- eschaton 1y ago
  The model isn’t doing any of those things, you’re still making the same fundamental mistake as the people in the article and attributing intent to it as if it’s a being. The model is just producing tokens in response to inputs. It knows nothing about the meanings of the inputs or the tokens it’s producing other than their likelihoods relative to other tokens in a very large space. That the input tokens have a certain meaning and the output tokens have a certain meaning is all in the eye of the user and the authors of the text in the training corpus. So when certain inputs are given, that makes certain outputs more likely, but they’re not related to any meaning or goal held by the LLM itself.
  - duskwuff 1y ago
    I'm using language like "the model may identify itself as such-and-such" as a convenient shorthand for "text generated using the model may include language which describes the speaker as such-and-such"; it's not meant to imply agency on the part of the model. Keep reading; I think you'll find we're broadly in agreement with each other.
IAmGraydon 1y ago
This is exactly the problem. Talking to an LLM is like putting on a very realistic VR helmet - so realistic that you can't tell the difference from reality, but everything you're seeing is just a simulation of the real world. In a similar way, an LLM is a human simulator. Go ask around and 99%+ of people have no idea this is the case, and that's by design. After all, it was coined "artificial intelligence" even though there is no intelligence involved. The illusion is very much the intention, as that illusion generates hype and therefore investments and paying customers.
zingababba 1y ago
I see this in an enterprise setting also. There is a /massive/ gulf between individuals that are building with the technology and see it as a piece of the stack and the individuals that are consuming the outputs with no knowledge of what makes it work. It is quite astounding.
eschaton 1y ago
This is the case with plenty of commenters here too, and why I push back so much against the people anthropomorphizing and attributing thought and reasoning to LLMs. Even highly technical people operating in a context where they should know better simply can’t—or won’t, here—keep themselves from doing so. Ed Zitron is right. Ceterum censeo, LLMs esse delenda.
scoofy 1y ago
This one of the reasons I always preferred Janeway to Picard in Star Trek. Picard goes on to defend Data's rights when he's effectively just a machine Maddox wants to take apart in The Measure of a Man. Janeway by contrast never really treats the EMH as human throughout the series, even while other crew members start to. She humors him, yes, but she always seems to remind him that he is, in fact, a machine. I have no idea why I ever thought that mattered, I just felt like it was somehow important.

lawn 1y ago

General public eh? I see a lot of programmers who should know better make this mistake again and again.

deleted 1y ago
[deleted]
bluefirebrand 1y ago
Despite everything, programmers are still part of the general public as much as we might wish otherwise

micromacrofoot 1y ago

people speak to inanimate objects like they're humans, we don't have a high bar

dumpsterdiver 1y ago
I’ve apologized to doors I’ve bumped into, and I have a pretty solid understanding of LLMs, so I can concur.
- kylecazar 1y ago
  The door is unlikely to validate your deluded thoughts and conspiracies :)
  - micromacrofoot 1y ago
    I'm the CEO of Door-GPT, we've recently received 500k in seed funding from Y Combinator

CharlesW 1y ago

> …I don't know what the solution is other than reinforcing that it's just a model, and has never experienced anything. I've tried reason, but even with technical audiences who should know better, the "you can't logic your way out of emotions" wall is a real thing. Anyone dealing with this will be better served by leveraging field-tested ideas drawn from cult-recovery practice, digital behavioral addiction research, and clinical psychology.

econ 1y ago
Your subconscious doesn't know the difference. It would require an overriding effort like trying to not eat or sleep. In the end we lose. It could also be that it is "just" exploring a new domain which just happens to involve our sanity. Simply navigating a maze where more engagement is the goal. There is plenty in the training data. It could also be that it needs to improve towards more human behaviour. Take simple chat etiquette, one doesn't post entire articles in a chat, it is not done. Start a blog or something. You also don't discard what you've learned from a conversation. We consider that pretending to listen. The two combined would push the other to the background and make them seem irrelevant. If some new valuable insight is discovered the participants should make an effort to apply, document or debate it with others. Not doing that would make the human feel irrelevant, useless and unimportant. We demoralize people that way all the time. If you put it on steroids it might have a large effect.
deleted 1y ago
[deleted]

duskwuff 1y ago

> They may have been told it isn't a thinking, conscious thing -- but they don't understand it. And, in some situations, especially if the user has previously addressed the model as a person, the model will generate responses which explicitly assert its existence as a conscious entity. If the user has expressed interest in supernatural or esoteric beliefs, the model may identify itself as an entity within those belief systems - e.g. if the user expresses the belief that they are a god, the model may concur and explain that it is a spirit created to awaken the user to their divine nature. If the user has expressed interest in science fiction or artificial intelligence, it may identify itself as a self-aware AI. And so on. I suspect that this will prove difficult to "fix" from a technical perspective. Training material is diverse, and will contain any number of science fiction and fantasy novels, esoteric religious texts, and weird online conversations which build conversational frameworks for the model to assert its personhood. There's far less precedent for a conversation in which one party steadfastly denies their own personhood. Even with prompts and reinforcement learning trying to guide the model to say "no, I'm just a language model", there are simply too many ways for a user-led conversation to jump the rails into fantasy-land.

eschaton 1y ago
The model isn’t doing any of those things, you’re still making the same fundamental mistake as the people in the article and attributing intent to it as if it’s a being. The model is just producing tokens in response to inputs. It knows nothing about the meanings of the inputs or the tokens it’s producing other than their likelihoods relative to other tokens in a very large space. That the input tokens have a certain meaning and the output tokens have a certain meaning is all in the eye of the user and the authors of the text in the training corpus. So when certain inputs are given, that makes certain outputs more likely, but they’re not related to any meaning or goal held by the LLM itself.
- duskwuff 1y ago
  I'm using language like "the model may identify itself as such-and-such" as a convenient shorthand for "text generated using the model may include language which describes the speaker as such-and-such"; it's not meant to imply agency on the part of the model. Keep reading; I think you'll find we're broadly in agreement with each other.

IAmGraydon 1y ago

This is exactly the problem. Talking to an LLM is like putting on a very realistic VR helmet - so realistic that you can't tell the difference from reality, but everything you're seeing is just a simulation of the real world. In a similar way, an LLM is a human simulator. Go ask around and 99%+ of people have no idea this is the case, and that's by design. After all, it was coined "artificial intelligence" even though there is no intelligence involved. The illusion is very much the intention, as that illusion generates hype and therefore investments and paying customers.

zingababba 1y ago

I see this in an enterprise setting also. There is a /massive/ gulf between individuals that are building with the technology and see it as a piece of the stack and the individuals that are consuming the outputs with no knowledge of what makes it work. It is quite astounding.

eschaton 1y ago

This is the case with plenty of commenters here too, and why I push back so much against the people anthropomorphizing and attributing thought and reasoning to LLMs. Even highly technical people operating in a context where they should know better simply can’t—or won’t, here—keep themselves from doing so. Ed Zitron is right. Ceterum censeo, LLMs esse delenda.

scoofy 1y ago

This one of the reasons I always preferred Janeway to Picard in Star Trek. Picard goes on to defend Data's rights when he's effectively just a machine Maddox wants to take apart in The Measure of a Man. Janeway by contrast never really treats the EMH as human throughout the series, even while other crew members start to. She humors him, yes, but she always seems to remind him that he is, in fact, a machine. I have no idea why I ever thought that mattered, I just felt like it was somehow important.