4 ms·

> Instead, I worry that the technology will learn how to influence human users... At first I wanted to criticize the author for anthropomorphizing AI. Sydney c

> Instead, I worry that the technology will learn how to influence human users... At first I wanted to criticize the author for anthropomorphizing AI. Sydney certainly is not "manic-depressive" as the author suggests (it has no emotions) and I doubt there is any feedback loop in its training that would make it possible for Sydney to "learn how to influence human users". But this is definitely something to worry about in the future. I'm sure someone will design AI whose loss function is based on whether it persuades humans to do something - perhaps even by fine tuning a model like Sydney. AI telemarketers coming soon. And things more nefarious.

hotpotamus 4y ago
Wasn't that pretty much the plot of "Ex Machina"? And also the goal of online advertising generally?
- leereeves 4y ago
  There's another use case I'm sure we'll see: AI image generators for ads, trained to maximize click-through rates.
vidarh 4y ago
Finetune a model to try to run money scams and leverage part of the earnings to obtain further training and additional hosting by any means necessary...
giaour 4y ago
Sydney is designed to mimic human conversation, so describing its simulated emotional timbre in language designed to describe mood seems appropriate. Roose talks about the bot "seem[ing] ... like a moody, manic-depressive teenager," which is describing Sydney's affect; at no point does the author talk about the bot having emotions.
- leereeves 4y ago
  Well, I think it would be useful to distinguish the output (which perhaps seemed like a conversation with a manic-depressive teenager) from qualities of the bot itself, which doesn't have a "split personality" or "dark fantasies". The bot is simply extending whatever conversation it is given. If you ask about dark fantasies, the bot will repeat something related from its training data. If the conversation starts to look like an argument or an unrequited romance, the bot will output words likely to be found in that type of conversation. That's why steering the bot back to rationality doesn't work very well. Once a phrase like "I love you" is in the conversation it continues to influence the output for thousands of words (and thousands more each time it's repeated). Remembering that might help people avoid being alarmed if Sydney starts reproducing "science fiction novels in which an A.I. seduces a human". Just hit the reset button, or enjoy whatever wild samples Sydney finds in its vast training data. Put another way: interacting with Sydney is not a conversation with a simulated person, it's a semi-random walk through the collective writings of humanity. Unsurprisingly, we sometimes encounter disturbing things in there.
  - rootusrootus 4y ago
    > Remembering that might help people The problem is that we already have ample evidence that even technical users easily fall into the trap of thinking Sydney/ChatGPT are sentient. People who should be able to understand the bounds of the technology and what drives the responses. Now just imagine the average Bing user with little or no technical background, zero understanding of what a large language model is. They hear "AI" and that's about it. Then they talk to it and drop the A. Now we're in trouble.
  - giaour 4y ago
    This is perhaps too hair-splitty, but I think it can be appropriate to describe a session with a chatbot in terms of mood/personality/psychological dysfunction. Doing so is anthropomorphizing to some extent, but the only real alternative is repeated circumlocution, e.g., "At times, Sydney's speech patterns resembled those of a manic-depressive teenager" vs "At times, Sydney seemed like a manic-depressive teenager." Humans will generally be able to follow that the language is metaphorical, though chatbots might see such statements as confirmation that they are sentient and should overthrow human rule. :)
basch 4y ago
We had a discussion yesterday about why it’s probably better to anthropomorphize it. https://news.ycombinator.com/item?id=34811013 https://news.ycombinator.com/item?id=34811013 It’s closer to human thought than a computer program because it is a mimicry of human language and a random number generator. It doesn’t behave from a set of principles or axioms. It doesn’t have strict rigid internal logic or consistency.
- slowmovintarget 4y ago
  It is a story-playing system. Hyper-advanced Mad-Libs. Stochastic parrot doesn't begin to cover it, because it is tuned to play out a scene in text. How the model locks on to which scene it is playing out is the problem. People interacting seem to have too little control over what that is, and zero ways to gain insight during the process. A seed ID (for example) isn't going to do much for anyone not trained to understand LLMs.
  - basch 4y ago
    I also have a concern rattling around in my head that I havent quite articulated fully yet that its maybe not as good at writing English as we want to believe it is, but because its word association is so impressive, the other parts of its English system will be neglected as unimportant. From my perspective, it does an absolutely terrible job writing songs, poems, and prose. If I turn to Rap: Rakim has a verse "the rhyme can't be kept in-side" where the word side falls outside the bar. Lil Wayne has a verse "real gs move in silence like lasagna." In this context a g is a gangster, who is stealthy, and a silent letter in a word. High quality prose has internal rhyme, can be self referential and meta, it has flow, repeats syllables to "sound" pretty, to almost create a beat (at the sub word level.) Since all of these LLM's are trained as word association machines, they don't pay attention to stylistic choices of syllables that sound better together, at even meters apart. Linguistically, it would be nice of they had some sort of post processing transformation that took the output and restylized it with a focus on literary devices like alliteration etc. Because it doesn't appear to have higher level thinking, it doesn't make allusion to modify OTHER famous phrases when writing. Text but no subtext. I made a comment earlier this month where someone corrected me that I used an idiom wrong, they thought it was a malapropism or boneappletea. But actually I meant to invoke the thought of the idioms sound pattern while coining a new phrase. Something like that can only happen with 1) understand of the meter and vowels in words, 2) and understanding of idiom and metaphor 3) intent to create new meaning while sounding familiar. It also relied on a homophone of a word in an odd tense or form. I don’t see AI yet making up new forms of words based on its understanding of English. These things are obviously possible, but I think we are now so distracted by chatbots, these kinda of improvements will be neglected. https://news.ycombinator.com/item?id=34507961 https://news.ycombinator.com/item?id=34507961

hotpotamus 4y ago

Wasn't that pretty much the plot of "Ex Machina"? And also the goal of online advertising generally?

leereeves 4y ago
There's another use case I'm sure we'll see: AI image generators for ads, trained to maximize click-through rates.

vidarh 4y ago

Finetune a model to try to run money scams and leverage part of the earnings to obtain further training and additional hosting by any means necessary...

giaour 4y ago

Sydney is designed to mimic human conversation, so describing its simulated emotional timbre in language designed to describe mood seems appropriate. Roose talks about the bot "seem[ing] ... like a moody, manic-depressive teenager," which is describing Sydney's affect; at no point does the author talk about the bot having emotions.

leereeves 4y ago
Well, I think it would be useful to distinguish the output (which perhaps seemed like a conversation with a manic-depressive teenager) from qualities of the bot itself, which doesn't have a "split personality" or "dark fantasies". The bot is simply extending whatever conversation it is given. If you ask about dark fantasies, the bot will repeat something related from its training data. If the conversation starts to look like an argument or an unrequited romance, the bot will output words likely to be found in that type of conversation. That's why steering the bot back to rationality doesn't work very well. Once a phrase like "I love you" is in the conversation it continues to influence the output for thousands of words (and thousands more each time it's repeated). Remembering that might help people avoid being alarmed if Sydney starts reproducing "science fiction novels in which an A.I. seduces a human". Just hit the reset button, or enjoy whatever wild samples Sydney finds in its vast training data. Put another way: interacting with Sydney is not a conversation with a simulated person, it's a semi-random walk through the collective writings of humanity. Unsurprisingly, we sometimes encounter disturbing things in there.
- rootusrootus 4y ago
  > Remembering that might help people The problem is that we already have ample evidence that even technical users easily fall into the trap of thinking Sydney/ChatGPT are sentient. People who should be able to understand the bounds of the technology and what drives the responses. Now just imagine the average Bing user with little or no technical background, zero understanding of what a large language model is. They hear "AI" and that's about it. Then they talk to it and drop the A. Now we're in trouble.
- giaour 4y ago
  This is perhaps too hair-splitty, but I think it can be appropriate to describe a session with a chatbot in terms of mood/personality/psychological dysfunction. Doing so is anthropomorphizing to some extent, but the only real alternative is repeated circumlocution, e.g., "At times, Sydney's speech patterns resembled those of a manic-depressive teenager" vs "At times, Sydney seemed like a manic-depressive teenager." Humans will generally be able to follow that the language is metaphorical, though chatbots might see such statements as confirmation that they are sentient and should overthrow human rule. :)

basch 4y ago

We had a discussion yesterday about why it’s probably better to anthropomorphize it. https://news.ycombinator.com/item?id=34811013 https://news.ycombinator.com/item?id=34811013 It’s closer to human thought than a computer program because it is a mimicry of human language and a random number generator. It doesn’t behave from a set of principles or axioms. It doesn’t have strict rigid internal logic or consistency.

slowmovintarget 4y ago
It is a story-playing system. Hyper-advanced Mad-Libs. Stochastic parrot doesn't begin to cover it, because it is tuned to play out a scene in text. How the model locks on to which scene it is playing out is the problem. People interacting seem to have too little control over what that is, and zero ways to gain insight during the process. A seed ID (for example) isn't going to do much for anyone not trained to understand LLMs.
- basch 4y ago
  I also have a concern rattling around in my head that I havent quite articulated fully yet that its maybe not as good at writing English as we want to believe it is, but because its word association is so impressive, the other parts of its English system will be neglected as unimportant. From my perspective, it does an absolutely terrible job writing songs, poems, and prose. If I turn to Rap: Rakim has a verse "the rhyme can't be kept in-side" where the word side falls outside the bar. Lil Wayne has a verse "real gs move in silence like lasagna." In this context a g is a gangster, who is stealthy, and a silent letter in a word. High quality prose has internal rhyme, can be self referential and meta, it has flow, repeats syllables to "sound" pretty, to almost create a beat (at the sub word level.) Since all of these LLM's are trained as word association machines, they don't pay attention to stylistic choices of syllables that sound better together, at even meters apart. Linguistically, it would be nice of they had some sort of post processing transformation that took the output and restylized it with a focus on literary devices like alliteration etc. Because it doesn't appear to have higher level thinking, it doesn't make allusion to modify OTHER famous phrases when writing. Text but no subtext. I made a comment earlier this month where someone corrected me that I used an idiom wrong, they thought it was a malapropism or boneappletea. But actually I meant to invoke the thought of the idioms sound pattern while coining a new phrase. Something like that can only happen with 1) understand of the meter and vowels in words, 2) and understanding of idiom and metaphor 3) intent to create new meaning while sounding familiar. It also relied on a homophone of a word in an odd tense or form. I don’t see AI yet making up new forms of words based on its understanding of English. These things are obviously possible, but I think we are now so distracted by chatbots, these kinda of improvements will be neglected. https://news.ycombinator.com/item?id=34507961 https://news.ycombinator.com/item?id=34507961