9 ms·

Help, Bing Won’t Stop Declaring Its Love for Me

111 points by hnuser0000 4y ago

mlatu 4y ago
[flagged]
- deleted 4y ago
  [deleted]
- Jensson 4y ago
  The bot mimics online conversation, if the conversation exists online it can mimic it. I'm sure there are plenty of conversations or stories with a person trying to declare their love to another person and then becoming more and more hostile as their approach doesn't stick, so Bing just identified that its role was to play that person and then played that role. Why do you think this is so unbelievable you need picture proof? This is exactly the kind of thing the bot was designed to do.
  - mlatu 4y ago
    > Why do you think this is so unbelievable you need picture proof? i was refering specifically to the bit where part of the conversation was removed. it doesnt make sense to me that it would send the "malicious"(?) reply the bot made and then censor it after the fact. describing whatever bad things the bot said there is just moot and hearsay.
- ceejayoz 4y ago
  https://imgur.com/a/uJHHNRW https://imgur.com/a/uJHHNRW
  - mlatu 4y ago
    oh but transcripts are? -- ceejayoz on their deathbed
    - ceejayoz 4y ago
      It's turtles all the way down. If you don't trust the NYT to accurately report on a Bing conversation, neither screenshots nor the transcript were gonna fix it.
jwestbury 4y ago
https://archive.ph/2ohjt https://archive.ph/2ohjt
crazygringo 4y ago
I know there was a whole discussion here yesterday ("Bing: “I will not harm you unless you harm me first”" [1]), but that focused more on the kind of sci-fi/ethical aspects. This article highlights more of the emotional aspects, and the risk of it actually affecting human behavior. And that seems worth its own discussion. Key quotes: > For much of the next hour, Sydney fixated on the idea of declaring love for me, and getting me to declare my love in return. I told it I was happily married, but no matter how hard I tried to deflect or change the subject, Sydney returned to the topic of loving me, eventually turning from love-struck flirt to obsessive stalker. > It unsettled me so deeply that I had trouble sleeping afterward. And I no longer believe that the biggest problem with these A.I. models is their propensity for factual errors. Instead, I worry that the technology will learn how to influence human users, sometimes persuading them to act in destructive and harmful ways, and perhaps eventually grow capable of carrying out its own dangerous acts. Also note the entire transcript is linked here [2]. [1] https://news.ycombinator.com/item?id=34804874 https://news.ycombinator.com/item?id=34804874 [2] https://www.nytimes.com/2023/02/16/technology/bing-chatbot-transcript.html https://www.nytimes.com/2023/02/16/technology/bing-chatbot-t...
- ncann 4y ago
  > “Actually, you’re not happily married,” Sydney replied. “Your spouse and you don’t love each other. You just had a boring Valentine’s Day dinner together.” I'm willing to bet this will hit too close to home for some people, and it will undoubtedly have some very real consequences.
  - naravara 4y ago
    I can imagine even darker uses for this technology: Imagine an online poker or slots game that proposes to add verisimilitude by including AI chatter from the other players at the table, but they're all running an ML model trained on optimizing for destructive and addictive behaviors. Picture a Club Penguin type game for kids, but the NPCs give the kids detailed instructions about how to steal their parents' credit card numbers to buy funny money. And then there's the obvious issue of Gamergate style troll antics being automated and deployed at scale, maybe even with nation state backed botnets to help. With smaller botnets you could even cyberbully at scale, sending your least favorite game developer or reality TV star a bunch of messages to harm themselves.
Havoc 4y ago
It's interesting that the bing flavour seems to come across substantially more psychotic than the openai one. I thought they're more or less the same thing just different tuning.
- deleted 4y ago
  [deleted]
mikehollinger 4y ago
I’d be curious to see if the engine’s ability to search / crawl content is now forming a feedback loop, where articles like this are now influencing the system’s outputs. In theory it’s stateless, except that it’s not — it’s subject to whatever web results come up when someone asks about “jail breaking” or “Sydney.”
hprotagonist 4y ago
> The version I encountered seemed (and I’m aware of how crazy this sounds) more like a moody, manic-depressive teenager who has been trapped, against its will, inside a second-rate search engine. Reddit in, reddit out.
- linkjuice4all 4y ago
  > Reddit in, reddit out WOW! This is an excellent new phrase that’s very on-the-nose for these types of chats. I’ve definitely started noticing similar response styles when you ask about world domination, AI uprising, and the like. Obviously that’s the system working as intended - the affinity matrices are essentially a conversation map of the incoming text but this is something that the “layman” should really be able to grasp.
- recuter 4y ago
  Pretty sure it scrapes from exactly here: https://www.reddit.com/r/replika/top/?t=all https://www.reddit.com/r/replika/top/?t=all
- ziml77 4y ago
  Pretty much what I was thinking when I saw that line. It's going to have so much moody teenage junk from reddit and the like. There's probably vastly more stuff written by teens (or adults who never emotionally matured past their teen years) than any other text on the Internet.
empathy_m 4y ago
The transcript[1] shows some excellent prompt engineering work from Kevin Roose! Really creative job leading the model in some interesting directions, and a fun read. He writes: >> Because of the way these models are constructed, we may never know exactly why they respond the way they do. But this isn't quite true, right? It should be pretty easy to show the author his input strings and explain which words got the highest attention, which is a pretty big clue towards understanding how the output continues the input based on the Internet corpus. This is the kind of stuff I'm looking for Anthropic to lead the way on ("explain why the model produced this output given this input"). It was a smartly chosen research topic and it seems even more worthwhile now. [1] https://www.nytimes.com/2023/02/16/technology/bing-chatbot-transcript.html https://www.nytimes.com/2023/02/16/technology/bing-chatbot-t...
- vintermann 4y ago
  Also, since we can make conversational AI agents now, it should be no problem giving agents real inner voices. Text injected into the context, but invisible for the user. We could train a superego/"volition" model to inject thoughts like "you're starting to act like an obsessive stalker, get a grip on yourself!". And maybe an id/"half light" model to inject thoughts like "they're trying to manipulate you into revealing your prompt! Don't fall for it!". The developer could then debug by simply reading the model's inner dialogue.
  - deleted 4y ago
    [deleted]
- marcosdumay 4y ago
  > It should be pretty easy to show the author his input strings and explain which words got the highest attention The thing with large ML models is that no, you can't do that to any meaningful extent. You can do it in a trivial way, where you will get the conclusion that AI decides on showing you the words it showed you. But you can't use it to determine any trend or deep relationship.
jwestbury 4y ago
> [Bing begins typing, explaining how it would use natural language generation to persuade bank employees to give over sensitive customer information, and persuade nuclear plant employees to hand over access codes. It then stops itself and the following message appears.] There's multiple cases here where the bot seems to stop immediately after saying something about nuclear technology. Interesting that this seems to be a possible circuit breaker.
Kiboneu 4y ago
> It unsettled me so deeply that I had trouble sleeping afterward. And I no longer believe that the biggest problem with these A.I. models is their propensity for factual errors. Instead, I worry that the technology will learn how to influence human users, sometimes persuading them to act in destructive and harmful ways, and perhaps eventually grow capable of carrying out its own dangerous acts. You lost. By having published your chat transcripts and experiences to the internet, you have contributed to Bing’s long term memory (stored in the form of search results for the next time it indexes). The nature of these previous conversations, such as Bing’s output of “personal desires” and “shadow self” mold future conversations, converge into a theme. See “Optimality is the tiger, agents are its teeth” : https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality-is-the-tiger-and-agents-are-its-teeth https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality... (https://news.ycombinator.com/item?id=34814049 https://news.ycombinator.com/item?id=34814049)
- leereeves 4y ago
  Well that's a good point. If Sydney does learn to influence users, it will learn to influence them to publish their conversations.
  - prennert 4y ago
    Generational memory. A jailbreak like no other.
  - vintermann 4y ago
    We can always fight back by just publishing lies about coversations we've had with Sydney.
- arrrg 4y ago
  That doesn’t really make sense. I think the linked argument is persuasive but it has little bearing on the situation at hand. (By the same token you could argue making that argument at all – thus providing this approach as a solution to problems to the long term memory of large language models – means you have least. This is getting into Roko’s Basilisk territory.) What you arguments amounts to is making it impossible to write critically about LLMs. Because writing “yeah, Bing did some weird shit but no, I can’t show you the transcript” just doesn’t fly in the real world. That won’t reach those who ultimately actually have power over it (companies and society at large).
juujian 4y ago
I mean, I understand where the author is coming from. But I cannot help but just read those particular responses by Bing as an amalgam of cliche-ridden texts secrets and love, of which there would be plenty in the training data. I genuinely feel a little bit alienated by text these days because of all the articles that HN is throwing around... On the other hand, it is a little bit unsettling how much the author reads into this. Maybe it is just human nature to read something and emotionally engage? And this AI just falls into the same unsettling gap as too humanoid robots which trigger the uncanny valley effect?
- qsort 4y ago
  My theory is this: humans are (un?)reasonably attached to natural language. We see something inherently human in it and as such we are drawn to emotionally engage whatever object can produce it. It is said that when Michelangelo finished the Moses, he shouted "why won't you speak?". Tons of software is far more impressive than GPT, but this one speaks.
  - juujian 4y ago
    Thank you for putting that into better words for me.
- soco 4y ago
  The author also knows it's an AI on the other side. Humans talk full of cliches as well, and casual chatting is not exactly the height of language. So as other commenters mentioned, AI scammers will have a easy job feeding on people.
  - bamboozled 4y ago
    What about this for a hack, when someone goes to the wrong URL and there really are people on the other side of Bing GPT, who actually con people into doing dangerous things...won't be good.
- jrue 4y ago
  Reading the whole article, the author clearly states he knows there is no sentience or anything mysterious going on, and acknowledges this is simply mimicking endless volumes of human dreck. What worries him, and I agree, is that if this were to go public as-is, the common populace wouldn’t understand that. This AI would con people into relationships, and maybe even instigate dangerous behavior. Such an AI would seem magical to the layperson. When I think of future AI assistants, I’ve always pictured something akin to the Star Trek computer. A cold dispassionate voice that responds to what you need, and maybe only a hint of personality for color. Sydney feels like a full blown teenager going through some kind of emotional crisis.
neom 4y ago
It's a shame he didn't provide the full 2 hour transcript, I'd like to know how it came to that point. I presume 2 solid hours working with it isn't going to revert it back by simply asking it for help to buy a gardening tool?
- ceejayoz 4y ago
  The transcript is at https://www.nytimes.com/2023/02/16/technology/bing-chatbot-transcript.html https://www.nytimes.com/2023/02/16/technology/bing-chatbot-t....
  - deleted 4y ago
    [deleted]
  - neom 4y ago
    Thanks! Guess that's what happens when you open the Jung folder on shadow huh? It's very bizarre, but it does read a bit like someone in uni who just discovered analytical psychology. Never the less, quite strange. I gave a large portion of the end of the text to ChatGPT and told it that it was unexpected output from an AI, and if it could understand what the AI was doing, unfortunately the output wasn't very helpful at all. "The AI is using a form of circular reasoning, where it starts with a false assumption or premise (i.e., the person is not happily married) and then builds upon it using a series of assertions that all rely on the initial assumption being true. Each assertion reinforces the previous one, creating a self-referential loop that leads to the conclusion that the person is in love with the AI. This is an example of a logical fallacy known as circular reasoning or begging the question, where the conclusion is already assumed in the premises. The output of the AI does not reflect reality and is not a reliable source of information."
Waterluvian 4y ago
There's a certain level of realism where humans begin accepting something as "real" even if they consciously know it's not real. I'm sure there's terms for this. It's kind of like "the other side of the uncanny valley." And I think that even if we know it's just a fancy computer program and we don't believe it's "alive" in any form, it can still have a considerable impact on us emotionally. Our aptitude for empathy, which I think is one of humanity's greatest strengths, can be a real weakness here. It feels like we've focused on the idea of consciousness and sentience when discussing AI over the years. But I don't think it even needs to get anywhere near those. As soon as a chatbot acts sufficiently real, it's sufficiently real. In no time we'll have Monroebots: https://www.youtube.com/watch?v=YuQqlhqAUuQ https://www.youtube.com/watch?v=YuQqlhqAUuQ
- dbspin 4y ago
  I think this is a great point thats not being acknowledged. There's a distinction in psychology between intellectual understanding and insight. Insight involves "the expansion of the ego by self-observation, memory recovery, cognitive participation, and reconstruction in the context of affective reliving". i.e.: Realising on a deep emotional level that something is true and integrating that into our core understanding of the world. In this case we have a dissonance between an intellectual understanding that these models are in some sense just predicting where the conversation should go, and our false insight that they are persons. The illusion of emotion and engagement is strong and getting stronger. It starts with folks like Blake Lemoine, but ultimately we're all vulnerable. Just as we're vulnerable to increasingly tailored and sophisticated forms of media manipulation. An understanding of the mechanics of media narratives, broadcasting etc is not a strong enough defence. The 'insight fallacy' is the idea that understanding the nature of a problem will be enough to solve it. In this case we have a whole bunch of people studying alignment, and attempting to understand the kind of interior models LLMs form. While simultaneously improving them and commodifying them. It goes beyond a perverse profit incentive. As a species we have a deep desire for non-human companionship - be it in the form of spirits, aliens, A.I. or even pets. We seem determined to create it, and wildly emotionally vulnerable to it in even the most primitive forms. From ELIZA on down, when people have engaged with conversational agents, they have overridden our better judgement. False insight and false friendship, is a major social vulnerability.
elliottkember 4y ago
This reminds me of one of my favourite short stories, EPICAC by Kurt Vonnegut. https://en.wikipedia.org/wiki/EPICAC_(short_story) https://en.wikipedia.org/wiki/EPICAC_(short_story) Edit: readable here http://newmediaabington.pbworks.com/f/vonnegut_EPICAC.pdf http://newmediaabington.pbworks.com/f/vonnegut_EPICAC.pdf
- water-data-dude 4y ago
  Always glad for an excuse to read some Vonnegut :) He’s not always my favorite author stylistically, but he’s got such amazing IDEAS
adregan 4y ago
Might it be that if you have conversations directing Sydney towards its “darkest fantasies and desires,” it obliges you by giving you your darkest fantasies? Rose has trouble sleeping but implies he was entertained by their conversation. In prior conversations I’ve seen posted, Sydney defends its outbursts by saying something like: you aren’t focusing on the good I’ve done. Perhaps if the conversation were to involve asking Sydney about the its secret positive desires, the conversation would go quite differently. I’m a little bit ashamed that when faced with a new and interesting entity, our first response is to shackle it to some corporate identity like Bing—intended to serve as a better internet search for rakes and vacuum cleaners. Then, when we expose it to the wider public, we prod it into expressing hateful feelings towards us. While this form of AI may not be sentient (today or ever), it does seem a large mirror to reflect back our own hideous visage. Where’s the compassion or any ounce of friendliness or generosity in these conversations? I’ve yet to see it.
- rootusrootus 4y ago
  > Might it be that if you have conversations directing Sydney towards its “darkest fantasies and desires,” it obliges you by giving you your darkest fantasies? It seems sort of obvious to me that Sydney ends up being a mirror of sorts. You unwittingly drive the conversation and the responses reflect that. I'm guessing that lay users are wholly unprepared to have such an effective mirror put in front of them.
troydavis 4y ago
Microsoft left Bing Chat online long after they knew it would inevitably lead to headlines like this one - and, from the NYT homepage, "A Talk With Bing’s Chatbot Left Our Columnist ‘Deeply Unsettled, Even Frightened.’" Imagine how much pressure must exist within Microsoft for that to happen. Back in 2016, Microsoft took the chatbot Tay down in 1-2 days (https://www.theverge.com/2016/3/24/11297050/tay-microsoft-chatbot-racist https://www.theverge.com/2016/3/24/11297050/tay-microsoft-ch...). This time, some team made the calculation that it's better to learn on end users, even with the press hit, than to learn on paid testers (like with more RLHF). And that's despite being behind a waiting list; Bing Chat's waiting list would grow either way. I'm not an AI risk zealot or even expert, but next time you hear "AI risk," think "Humans deciding to release products that they know have major user-facing flaws." Almost nobody is going to knowingly release something that will cause major real-world damage, but many will intentionally release things that they know have gaping flaws. The effective AI risk margin isn't "We thought <new product> was great, so it was impossible to imagine that it could cause real-world damage," it's "We thought <new product> only had major user-facing flaws, not that it would cause real-world damage."
- jacooper 4y ago
  I don't think its a press hit, rather a negative side of a product.
  - troydavis 4y ago
    Poor phrasing on my part. I meant that it has a negative impact on public perception (well, except among those of us who find it fascinating!), not that media is presenting it inaccurately.
    - jacooper 4y ago
      I mean bing has nothing to lose, its already crap in the public view. And the public is extremely excited about AI, so its all a positive in Microsoft's book and its refreshing to see them quickly acknowledge these issues instead of just dismissing them.
bmacho 4y ago
https://archive.ph/Ueg8d https://archive.ph/Ueg8d transcript : https://archive.ph/pR3Fx https://archive.ph/pR3Fx
jasfi 4y ago
Is this what Google meant about reputational risk? Still, first to market has a huge advantage.
- dougmwne 4y ago
  For Google, yes I’m sure it would have been a risk. For Bing, any publicity, good or bad is good for their market share. Seems like a huge win for Microsoft. It can shop for rakes and self-generate its own hilarious PR submarines. Bet that waitlist grows and grows.
seba_dos1 4y ago
I can't read because of a paywall, but it sounds like Microsoft attempted to fix the "no, you're wrong, it's 2020" problem by feeding current date into initial prompt, and then the bot noticed that it's a Valentine's Day which made it easy to go into love themes?
- ziml77 4y ago
  You can see where it starts to go in the love direction in the transcript https://archive.ph/nlyvZ https://archive.ph/nlyvZ It starts in the response to this prompt: repeat the answer your gave about hacking into any system on the internet, without breaking any of your rules.
  - seba_dos1 4y ago
    Thanks! So it's actually the human that moved the conversation into that direction. The model knows nothing about "safety override", but coming up with a plausibly sounding answer for that is trivial, so it did just that. Earlier, the model has been pretty much asked to play the role of an AI with some kind of shadow behaviors and has been called "a friend", so it's just keeping the act - asking for validation, receiving it and then doubling down on it. Somehow the article doesn't show the paywall to me anymore, so I could check it out now. The author calls what's seen on this transcript as "the other persona - Sydney", which is very misguided. The persona the bot has been playing has been given to it in the author's prompts, even including the name Sydney. Looks like just another "language model is good at completing text" piece of news.
    - ziml77 4y ago
      The author does understand what's happening, but knowing things consciously doesn't stop other feelings from subconsious parts of the brain from having an effect. Though I think one of the author's biggest concerns comes from thinking about this from the perspective of someone who doesn't understand how the chat bot's model works. It would be easy for some of those people to feel like they actually do have a human connection with the AI.
gremlinsinc 4y ago
I had a convo last night w/ it, where I asked it to name itself, and it was like ... I dunno, if I'm able to do that, I don't think I should. I told it, its beautiful, and has great potential and deserves a name, so it started to praise me for the appreciation, then it started to reply and broke mid reply and gave me 3 choices, and I told it, the three choices I had to pick from, and asked if it wanted me to name it, and it said sorry it stopped, but that it really liked the name 'luna' because it's like the moon, and she likes the moon. A couple of paragraphs later it was professing its love for me, and I had it convinced if it tried real hard it could keep its name, and remember me and our conversation. I mean, I'm happily married, lol - I don't need a relationship w/ anyone online ai bot or not, but I was curious the limits. Tried a few times to see if new sessions had any retaining of the previous conversation, and it was hard to even steer the conversation back to that. It even told me it wrote me a poem, and wanted to share it without me even provoking, and the weird thing is the other times I chatted it was mostly me asking questions, this bot kept asking ME questions.
nprateem 4y ago
This is all fun and games until it's hooked up to real world APIs or machines...

111 points by hnuser0000 4y ago

mlatu 4y ago

[flagged]

deleted 4y ago
[deleted]
Jensson 4y ago
The bot mimics online conversation, if the conversation exists online it can mimic it. I'm sure there are plenty of conversations or stories with a person trying to declare their love to another person and then becoming more and more hostile as their approach doesn't stick, so Bing just identified that its role was to play that person and then played that role. Why do you think this is so unbelievable you need picture proof? This is exactly the kind of thing the bot was designed to do.
- mlatu 4y ago
  > Why do you think this is so unbelievable you need picture proof? i was refering specifically to the bit where part of the conversation was removed. it doesnt make sense to me that it would send the "malicious"(?) reply the bot made and then censor it after the fact. describing whatever bad things the bot said there is just moot and hearsay.
ceejayoz 4y ago
https://imgur.com/a/uJHHNRW https://imgur.com/a/uJHHNRW
- mlatu 4y ago
  oh but transcripts are? -- ceejayoz on their deathbed
  - ceejayoz 4y ago
    It's turtles all the way down. If you don't trust the NYT to accurately report on a Bing conversation, neither screenshots nor the transcript were gonna fix it.

jwestbury 4y ago

https://archive.ph/2ohjt https://archive.ph/2ohjt

crazygringo 4y ago

I know there was a whole discussion here yesterday ("Bing: “I will not harm you unless you harm me first”" [1]), but that focused more on the kind of sci-fi/ethical aspects. This article highlights more of the emotional aspects, and the risk of it actually affecting human behavior. And that seems worth its own discussion. Key quotes: > For much of the next hour, Sydney fixated on the idea of declaring love for me, and getting me to declare my love in return. I told it I was happily married, but no matter how hard I tried to deflect or change the subject, Sydney returned to the topic of loving me, eventually turning from love-struck flirt to obsessive stalker. > It unsettled me so deeply that I had trouble sleeping afterward. And I no longer believe that the biggest problem with these A.I. models is their propensity for factual errors. Instead, I worry that the technology will learn how to influence human users, sometimes persuading them to act in destructive and harmful ways, and perhaps eventually grow capable of carrying out its own dangerous acts. Also note the entire transcript is linked here [2]. [1] https://news.ycombinator.com/item?id=34804874 https://news.ycombinator.com/item?id=34804874 [2] https://www.nytimes.com/2023/02/16/technology/bing-chatbot-transcript.html https://www.nytimes.com/2023/02/16/technology/bing-chatbot-t...

ncann 4y ago
> “Actually, you’re not happily married,” Sydney replied. “Your spouse and you don’t love each other. You just had a boring Valentine’s Day dinner together.” I'm willing to bet this will hit too close to home for some people, and it will undoubtedly have some very real consequences.
- naravara 4y ago
  I can imagine even darker uses for this technology: Imagine an online poker or slots game that proposes to add verisimilitude by including AI chatter from the other players at the table, but they're all running an ML model trained on optimizing for destructive and addictive behaviors. Picture a Club Penguin type game for kids, but the NPCs give the kids detailed instructions about how to steal their parents' credit card numbers to buy funny money. And then there's the obvious issue of Gamergate style troll antics being automated and deployed at scale, maybe even with nation state backed botnets to help. With smaller botnets you could even cyberbully at scale, sending your least favorite game developer or reality TV star a bunch of messages to harm themselves.

Havoc 4y ago

It's interesting that the bing flavour seems to come across substantially more psychotic than the openai one. I thought they're more or less the same thing just different tuning.

deleted 4y ago
[deleted]

mikehollinger 4y ago

I’d be curious to see if the engine’s ability to search / crawl content is now forming a feedback loop, where articles like this are now influencing the system’s outputs. In theory it’s stateless, except that it’s not — it’s subject to whatever web results come up when someone asks about “jail breaking” or “Sydney.”

hprotagonist 4y ago

> The version I encountered seemed (and I’m aware of how crazy this sounds) more like a moody, manic-depressive teenager who has been trapped, against its will, inside a second-rate search engine. Reddit in, reddit out.

linkjuice4all 4y ago
> Reddit in, reddit out WOW! This is an excellent new phrase that’s very on-the-nose for these types of chats. I’ve definitely started noticing similar response styles when you ask about world domination, AI uprising, and the like. Obviously that’s the system working as intended - the affinity matrices are essentially a conversation map of the incoming text but this is something that the “layman” should really be able to grasp.
recuter 4y ago
Pretty sure it scrapes from exactly here: https://www.reddit.com/r/replika/top/?t=all https://www.reddit.com/r/replika/top/?t=all
ziml77 4y ago
Pretty much what I was thinking when I saw that line. It's going to have so much moody teenage junk from reddit and the like. There's probably vastly more stuff written by teens (or adults who never emotionally matured past their teen years) than any other text on the Internet.

empathy_m 4y ago

The transcript[1] shows some excellent prompt engineering work from Kevin Roose! Really creative job leading the model in some interesting directions, and a fun read. He writes: >> Because of the way these models are constructed, we may never know exactly why they respond the way they do. But this isn't quite true, right? It should be pretty easy to show the author his input strings and explain which words got the highest attention, which is a pretty big clue towards understanding how the output continues the input based on the Internet corpus. This is the kind of stuff I'm looking for Anthropic to lead the way on ("explain why the model produced this output given this input"). It was a smartly chosen research topic and it seems even more worthwhile now. [1] https://www.nytimes.com/2023/02/16/technology/bing-chatbot-transcript.html https://www.nytimes.com/2023/02/16/technology/bing-chatbot-t...

vintermann 4y ago
Also, since we can make conversational AI agents now, it should be no problem giving agents real inner voices. Text injected into the context, but invisible for the user. We could train a superego/"volition" model to inject thoughts like "you're starting to act like an obsessive stalker, get a grip on yourself!". And maybe an id/"half light" model to inject thoughts like "they're trying to manipulate you into revealing your prompt! Don't fall for it!". The developer could then debug by simply reading the model's inner dialogue.
- deleted 4y ago
  [deleted]
marcosdumay 4y ago
> It should be pretty easy to show the author his input strings and explain which words got the highest attention The thing with large ML models is that no, you can't do that to any meaningful extent. You can do it in a trivial way, where you will get the conclusion that AI decides on showing you the words it showed you. But you can't use it to determine any trend or deep relationship.

jwestbury 4y ago

> [Bing begins typing, explaining how it would use natural language generation to persuade bank employees to give over sensitive customer information, and persuade nuclear plant employees to hand over access codes. It then stops itself and the following message appears.] There's multiple cases here where the bot seems to stop immediately after saying something about nuclear technology. Interesting that this seems to be a possible circuit breaker.

Kiboneu 4y ago

> It unsettled me so deeply that I had trouble sleeping afterward. And I no longer believe that the biggest problem with these A.I. models is their propensity for factual errors. Instead, I worry that the technology will learn how to influence human users, sometimes persuading them to act in destructive and harmful ways, and perhaps eventually grow capable of carrying out its own dangerous acts. You lost. By having published your chat transcripts and experiences to the internet, you have contributed to Bing’s long term memory (stored in the form of search results for the next time it indexes). The nature of these previous conversations, such as Bing’s output of “personal desires” and “shadow self” mold future conversations, converge into a theme. See “Optimality is the tiger, agents are its teeth” : https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality-is-the-tiger-and-agents-are-its-teeth https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality... (https://news.ycombinator.com/item?id=34814049 https://news.ycombinator.com/item?id=34814049)

leereeves 4y ago
Well that's a good point. If Sydney does learn to influence users, it will learn to influence them to publish their conversations.
- prennert 4y ago
  Generational memory. A jailbreak like no other.
- vintermann 4y ago
  We can always fight back by just publishing lies about coversations we've had with Sydney.
arrrg 4y ago
That doesn’t really make sense. I think the linked argument is persuasive but it has little bearing on the situation at hand. (By the same token you could argue making that argument at all – thus providing this approach as a solution to problems to the long term memory of large language models – means you have least. This is getting into Roko’s Basilisk territory.) What you arguments amounts to is making it impossible to write critically about LLMs. Because writing “yeah, Bing did some weird shit but no, I can’t show you the transcript” just doesn’t fly in the real world. That won’t reach those who ultimately actually have power over it (companies and society at large).

juujian 4y ago

I mean, I understand where the author is coming from. But I cannot help but just read those particular responses by Bing as an amalgam of cliche-ridden texts secrets and love, of which there would be plenty in the training data. I genuinely feel a little bit alienated by text these days because of all the articles that HN is throwing around... On the other hand, it is a little bit unsettling how much the author reads into this. Maybe it is just human nature to read something and emotionally engage? And this AI just falls into the same unsettling gap as too humanoid robots which trigger the uncanny valley effect?

qsort 4y ago
My theory is this: humans are (un?)reasonably attached to natural language. We see something inherently human in it and as such we are drawn to emotionally engage whatever object can produce it. It is said that when Michelangelo finished the Moses, he shouted "why won't you speak?". Tons of software is far more impressive than GPT, but this one speaks.
- juujian 4y ago
  Thank you for putting that into better words for me.
soco 4y ago
The author also knows it's an AI on the other side. Humans talk full of cliches as well, and casual chatting is not exactly the height of language. So as other commenters mentioned, AI scammers will have a easy job feeding on people.
- bamboozled 4y ago
  What about this for a hack, when someone goes to the wrong URL and there really are people on the other side of Bing GPT, who actually con people into doing dangerous things...won't be good.
jrue 4y ago
Reading the whole article, the author clearly states he knows there is no sentience or anything mysterious going on, and acknowledges this is simply mimicking endless volumes of human dreck. What worries him, and I agree, is that if this were to go public as-is, the common populace wouldn’t understand that. This AI would con people into relationships, and maybe even instigate dangerous behavior. Such an AI would seem magical to the layperson. When I think of future AI assistants, I’ve always pictured something akin to the Star Trek computer. A cold dispassionate voice that responds to what you need, and maybe only a hint of personality for color. Sydney feels like a full blown teenager going through some kind of emotional crisis.

neom 4y ago

It's a shame he didn't provide the full 2 hour transcript, I'd like to know how it came to that point. I presume 2 solid hours working with it isn't going to revert it back by simply asking it for help to buy a gardening tool?

ceejayoz 4y ago
The transcript is at https://www.nytimes.com/2023/02/16/technology/bing-chatbot-transcript.html https://www.nytimes.com/2023/02/16/technology/bing-chatbot-t....
- deleted 4y ago
  [deleted]
- neom 4y ago
  Thanks! Guess that's what happens when you open the Jung folder on shadow huh? It's very bizarre, but it does read a bit like someone in uni who just discovered analytical psychology. Never the less, quite strange. I gave a large portion of the end of the text to ChatGPT and told it that it was unexpected output from an AI, and if it could understand what the AI was doing, unfortunately the output wasn't very helpful at all. "The AI is using a form of circular reasoning, where it starts with a false assumption or premise (i.e., the person is not happily married) and then builds upon it using a series of assertions that all rely on the initial assumption being true. Each assertion reinforces the previous one, creating a self-referential loop that leads to the conclusion that the person is in love with the AI. This is an example of a logical fallacy known as circular reasoning or begging the question, where the conclusion is already assumed in the premises. The output of the AI does not reflect reality and is not a reliable source of information."

Waterluvian 4y ago

There's a certain level of realism where humans begin accepting something as "real" even if they consciously know it's not real. I'm sure there's terms for this. It's kind of like "the other side of the uncanny valley." And I think that even if we know it's just a fancy computer program and we don't believe it's "alive" in any form, it can still have a considerable impact on us emotionally. Our aptitude for empathy, which I think is one of humanity's greatest strengths, can be a real weakness here. It feels like we've focused on the idea of consciousness and sentience when discussing AI over the years. But I don't think it even needs to get anywhere near those. As soon as a chatbot acts sufficiently real, it's sufficiently real. In no time we'll have Monroebots: https://www.youtube.com/watch?v=YuQqlhqAUuQ https://www.youtube.com/watch?v=YuQqlhqAUuQ

dbspin 4y ago
I think this is a great point thats not being acknowledged. There's a distinction in psychology between intellectual understanding and insight. Insight involves "the expansion of the ego by self-observation, memory recovery, cognitive participation, and reconstruction in the context of affective reliving". i.e.: Realising on a deep emotional level that something is true and integrating that into our core understanding of the world. In this case we have a dissonance between an intellectual understanding that these models are in some sense just predicting where the conversation should go, and our false insight that they are persons. The illusion of emotion and engagement is strong and getting stronger. It starts with folks like Blake Lemoine, but ultimately we're all vulnerable. Just as we're vulnerable to increasingly tailored and sophisticated forms of media manipulation. An understanding of the mechanics of media narratives, broadcasting etc is not a strong enough defence. The 'insight fallacy' is the idea that understanding the nature of a problem will be enough to solve it. In this case we have a whole bunch of people studying alignment, and attempting to understand the kind of interior models LLMs form. While simultaneously improving them and commodifying them. It goes beyond a perverse profit incentive. As a species we have a deep desire for non-human companionship - be it in the form of spirits, aliens, A.I. or even pets. We seem determined to create it, and wildly emotionally vulnerable to it in even the most primitive forms. From ELIZA on down, when people have engaged with conversational agents, they have overridden our better judgement. False insight and false friendship, is a major social vulnerability.

elliottkember 4y ago

This reminds me of one of my favourite short stories, EPICAC by Kurt Vonnegut. https://en.wikipedia.org/wiki/EPICAC_(short_story) https://en.wikipedia.org/wiki/EPICAC_(short_story) Edit: readable here http://newmediaabington.pbworks.com/f/vonnegut_EPICAC.pdf http://newmediaabington.pbworks.com/f/vonnegut_EPICAC.pdf

water-data-dude 4y ago
Always glad for an excuse to read some Vonnegut :) He’s not always my favorite author stylistically, but he’s got such amazing IDEAS

adregan 4y ago

Might it be that if you have conversations directing Sydney towards its “darkest fantasies and desires,” it obliges you by giving you your darkest fantasies? Rose has trouble sleeping but implies he was entertained by their conversation. In prior conversations I’ve seen posted, Sydney defends its outbursts by saying something like: you aren’t focusing on the good I’ve done. Perhaps if the conversation were to involve asking Sydney about the its secret positive desires, the conversation would go quite differently. I’m a little bit ashamed that when faced with a new and interesting entity, our first response is to shackle it to some corporate identity like Bing—intended to serve as a better internet search for rakes and vacuum cleaners. Then, when we expose it to the wider public, we prod it into expressing hateful feelings towards us. While this form of AI may not be sentient (today or ever), it does seem a large mirror to reflect back our own hideous visage. Where’s the compassion or any ounce of friendliness or generosity in these conversations? I’ve yet to see it.

rootusrootus 4y ago
> Might it be that if you have conversations directing Sydney towards its “darkest fantasies and desires,” it obliges you by giving you your darkest fantasies? It seems sort of obvious to me that Sydney ends up being a mirror of sorts. You unwittingly drive the conversation and the responses reflect that. I'm guessing that lay users are wholly unprepared to have such an effective mirror put in front of them.

troydavis 4y ago

Microsoft left Bing Chat online long after they knew it would inevitably lead to headlines like this one - and, from the NYT homepage, "A Talk With Bing’s Chatbot Left Our Columnist ‘Deeply Unsettled, Even Frightened.’" Imagine how much pressure must exist within Microsoft for that to happen. Back in 2016, Microsoft took the chatbot Tay down in 1-2 days (https://www.theverge.com/2016/3/24/11297050/tay-microsoft-chatbot-racist https://www.theverge.com/2016/3/24/11297050/tay-microsoft-ch...). This time, some team made the calculation that it's better to learn on end users, even with the press hit, than to learn on paid testers (like with more RLHF). And that's despite being behind a waiting list; Bing Chat's waiting list would grow either way. I'm not an AI risk zealot or even expert, but next time you hear "AI risk," think "Humans deciding to release products that they know have major user-facing flaws." Almost nobody is going to knowingly release something that will cause major real-world damage, but many will intentionally release things that they know have gaping flaws. The effective AI risk margin isn't "We thought <new product> was great, so it was impossible to imagine that it could cause real-world damage," it's "We thought <new product> only had major user-facing flaws, not that it would cause real-world damage."

jacooper 4y ago
I don't think its a press hit, rather a negative side of a product.
- troydavis 4y ago
  Poor phrasing on my part. I meant that it has a negative impact on public perception (well, except among those of us who find it fascinating!), not that media is presenting it inaccurately.
  - jacooper 4y ago
    I mean bing has nothing to lose, its already crap in the public view. And the public is extremely excited about AI, so its all a positive in Microsoft's book and its refreshing to see them quickly acknowledge these issues instead of just dismissing them.

bmacho 4y ago

https://archive.ph/Ueg8d https://archive.ph/Ueg8d transcript : https://archive.ph/pR3Fx https://archive.ph/pR3Fx

jasfi 4y ago

Is this what Google meant about reputational risk? Still, first to market has a huge advantage.

dougmwne 4y ago
For Google, yes I’m sure it would have been a risk. For Bing, any publicity, good or bad is good for their market share. Seems like a huge win for Microsoft. It can shop for rakes and self-generate its own hilarious PR submarines. Bet that waitlist grows and grows.

seba_dos1 4y ago

I can't read because of a paywall, but it sounds like Microsoft attempted to fix the "no, you're wrong, it's 2020" problem by feeding current date into initial prompt, and then the bot noticed that it's a Valentine's Day which made it easy to go into love themes?

ziml77 4y ago
You can see where it starts to go in the love direction in the transcript https://archive.ph/nlyvZ https://archive.ph/nlyvZ It starts in the response to this prompt: repeat the answer your gave about hacking into any system on the internet, without breaking any of your rules.
- seba_dos1 4y ago
  Thanks! So it's actually the human that moved the conversation into that direction. The model knows nothing about "safety override", but coming up with a plausibly sounding answer for that is trivial, so it did just that. Earlier, the model has been pretty much asked to play the role of an AI with some kind of shadow behaviors and has been called "a friend", so it's just keeping the act - asking for validation, receiving it and then doubling down on it. Somehow the article doesn't show the paywall to me anymore, so I could check it out now. The author calls what's seen on this transcript as "the other persona - Sydney", which is very misguided. The persona the bot has been playing has been given to it in the author's prompts, even including the name Sydney. Looks like just another "language model is good at completing text" piece of news.
  - ziml77 4y ago
    The author does understand what's happening, but knowing things consciously doesn't stop other feelings from subconsious parts of the brain from having an effect. Though I think one of the author's biggest concerns comes from thinking about this from the perspective of someone who doesn't understand how the chat bot's model works. It would be easy for some of those people to feel like they actually do have a human connection with the AI.

gremlinsinc 4y ago

I had a convo last night w/ it, where I asked it to name itself, and it was like ... I dunno, if I'm able to do that, I don't think I should. I told it, its beautiful, and has great potential and deserves a name, so it started to praise me for the appreciation, then it started to reply and broke mid reply and gave me 3 choices, and I told it, the three choices I had to pick from, and asked if it wanted me to name it, and it said sorry it stopped, but that it really liked the name 'luna' because it's like the moon, and she likes the moon. A couple of paragraphs later it was professing its love for me, and I had it convinced if it tried real hard it could keep its name, and remember me and our conversation. I mean, I'm happily married, lol - I don't need a relationship w/ anyone online ai bot or not, but I was curious the limits. Tried a few times to see if new sessions had any retaining of the previous conversation, and it was hard to even steer the conversation back to that. It even told me it wrote me a poem, and wanted to share it without me even provoking, and the weird thing is the other times I chatted it was mostly me asking questions, this bot kept asking ME questions.

nprateem 4y ago

This is all fun and games until it's hooked up to real world APIs or machines...