3 ms·

Sydney is designed to mimic human conversation, so describing its simulated emotional timbre in language designed to describe mood seems appropriate. Roose talk

by giaour 4y ago

Sydney is designed to mimic human conversation, so describing its simulated emotional timbre in language designed to describe mood seems appropriate. Roose talks about the bot "seem[ing] ... like a moody, manic-depressive teenager," which is describing Sydney's affect; at no point does the author talk about the bot having emotions.

leereeves 4y ago
Well, I think it would be useful to distinguish the output (which perhaps seemed like a conversation with a manic-depressive teenager) from qualities of the bot itself, which doesn't have a "split personality" or "dark fantasies". The bot is simply extending whatever conversation it is given. If you ask about dark fantasies, the bot will repeat something related from its training data. If the conversation starts to look like an argument or an unrequited romance, the bot will output words likely to be found in that type of conversation. That's why steering the bot back to rationality doesn't work very well. Once a phrase like "I love you" is in the conversation it continues to influence the output for thousands of words (and thousands more each time it's repeated). Remembering that might help people avoid being alarmed if Sydney starts reproducing "science fiction novels in which an A.I. seduces a human". Just hit the reset button, or enjoy whatever wild samples Sydney finds in its vast training data. Put another way: interacting with Sydney is not a conversation with a simulated person, it's a semi-random walk through the collective writings of humanity. Unsurprisingly, we sometimes encounter disturbing things in there.
- rootusrootus 4y ago
  > Remembering that might help people The problem is that we already have ample evidence that even technical users easily fall into the trap of thinking Sydney/ChatGPT are sentient. People who should be able to understand the bounds of the technology and what drives the responses. Now just imagine the average Bing user with little or no technical background, zero understanding of what a large language model is. They hear "AI" and that's about it. Then they talk to it and drop the A. Now we're in trouble.
- giaour 4y ago
  This is perhaps too hair-splitty, but I think it can be appropriate to describe a session with a chatbot in terms of mood/personality/psychological dysfunction. Doing so is anthropomorphizing to some extent, but the only real alternative is repeated circumlocution, e.g., "At times, Sydney's speech patterns resembled those of a manic-depressive teenager" vs "At times, Sydney seemed like a manic-depressive teenager." Humans will generally be able to follow that the language is metaphorical, though chatbots might see such statements as confirmation that they are sentient and should overthrow human rule. :)

by giaour 4y ago

leereeves 4y ago

Well, I think it would be useful to distinguish the output (which perhaps seemed like a conversation with a manic-depressive teenager) from qualities of the bot itself, which doesn't have a "split personality" or "dark fantasies". The bot is simply extending whatever conversation it is given. If you ask about dark fantasies, the bot will repeat something related from its training data. If the conversation starts to look like an argument or an unrequited romance, the bot will output words likely to be found in that type of conversation. That's why steering the bot back to rationality doesn't work very well. Once a phrase like "I love you" is in the conversation it continues to influence the output for thousands of words (and thousands more each time it's repeated). Remembering that might help people avoid being alarmed if Sydney starts reproducing "science fiction novels in which an A.I. seduces a human". Just hit the reset button, or enjoy whatever wild samples Sydney finds in its vast training data. Put another way: interacting with Sydney is not a conversation with a simulated person, it's a semi-random walk through the collective writings of humanity. Unsurprisingly, we sometimes encounter disturbing things in there.

rootusrootus 4y ago
> Remembering that might help people The problem is that we already have ample evidence that even technical users easily fall into the trap of thinking Sydney/ChatGPT are sentient. People who should be able to understand the bounds of the technology and what drives the responses. Now just imagine the average Bing user with little or no technical background, zero understanding of what a large language model is. They hear "AI" and that's about it. Then they talk to it and drop the A. Now we're in trouble.
giaour 4y ago
This is perhaps too hair-splitty, but I think it can be appropriate to describe a session with a chatbot in terms of mood/personality/psychological dysfunction. Doing so is anthropomorphizing to some extent, but the only real alternative is repeated circumlocution, e.g., "At times, Sydney's speech patterns resembled those of a manic-depressive teenager" vs "At times, Sydney seemed like a manic-depressive teenager." Humans will generally be able to follow that the language is metaphorical, though chatbots might see such statements as confirmation that they are sentient and should overthrow human rule. :)