3 ms·

i found it weird how LLMs will say "i" and "my", i tried to ask it about whether this implied it had some concept of self, and then does it also have its own op

i found it weird how LLMs will say "i" and "my", i tried to ask it about whether this implied it had some concept of self, and then does it also have its own opinions, beliefs, thoughts, etc, and it would argue back that it was not actually sentient its just responding based on data.

taneq 3y ago
That's because it's been fine tuned on RLHF data which gives those responses to that kind of question. It says "I" because that's how people talk and it's modeling how people talk. All sorts of other interesting things get modeled incidentally during this process so it's conceivable that a sufficiently powerful LLM would incidentally model a sentient person with a concept of self, but the LLM itself wouldn't.
- doubloon 3y ago
  so if you have an entity that fakes all the inputs and outputs that would indicate that it has a concept of self, how can you tell the difference between that and a entity that actually does have a sense of self?
  - taneq 3y ago
    If this entity successfully faked all those outputs then you wouldn't be able to tell the difference, by definition, since if you could tell the difference then they wouldn't be successfully faked. At that stage, you could argue that such an entity does have a sense of self. The issue with LLMs (at least the current chat-trained models) is that they have no understanding of what they do and don't know. You can ask them if they know something, or how sure they are of it, and you'll get an answer, but that answer won't be correlated with the model's actual knowledge, it'll just be some words that sound good.
ants_everywhere 3y ago
People seem to forget this because it happened before ChatGPT, but a Google engineer convinced himself that the predecessor of Bard was self-aware. https://www.scientificamerican.com/article/google-engineer-claims-ai-chatbot-is-sentient-why-that-matters/ https://www.scientificamerican.com/article/google-engineer-c... The AI mentioned is LaMDA, which according to this blog post powers Bard: https://blog.google/technology/ai/bard-google-ai-search-updates/ https://blog.google/technology/ai/bard-google-ai-search-upda...

taneq 3y ago

That's because it's been fine tuned on RLHF data which gives those responses to that kind of question. It says "I" because that's how people talk and it's modeling how people talk. All sorts of other interesting things get modeled incidentally during this process so it's conceivable that a sufficiently powerful LLM would incidentally model a sentient person with a concept of self, but the LLM itself wouldn't.

doubloon 3y ago
so if you have an entity that fakes all the inputs and outputs that would indicate that it has a concept of self, how can you tell the difference between that and a entity that actually does have a sense of self?
- taneq 3y ago
  If this entity successfully faked all those outputs then you wouldn't be able to tell the difference, by definition, since if you could tell the difference then they wouldn't be successfully faked. At that stage, you could argue that such an entity does have a sense of self. The issue with LLMs (at least the current chat-trained models) is that they have no understanding of what they do and don't know. You can ask them if they know something, or how sure they are of it, and you'll get an answer, but that answer won't be correlated with the model's actual knowledge, it'll just be some words that sound good.

ants_everywhere 3y ago

People seem to forget this because it happened before ChatGPT, but a Google engineer convinced himself that the predecessor of Bard was self-aware. https://www.scientificamerican.com/article/google-engineer-claims-ai-chatbot-is-sentient-why-that-matters/ https://www.scientificamerican.com/article/google-engineer-c... The AI mentioned is LaMDA, which according to this blog post powers Bard: https://blog.google/technology/ai/bard-google-ai-search-updates/ https://blog.google/technology/ai/bard-google-ai-search-upda...