13 ms·

They link to the complaint, which is obviously a lot longer than the single message [0]. The child, J.F., is autistic and has allegedly exhibited a spiralling t

They link to the complaint, which is obviously a lot longer than the single message [0]. The child, J.F., is autistic and has allegedly exhibited a spiralling trend of aggressive behavior towards his parents which they attribute to the content fed by the Character AI app: > Only then did she discover J.F.’s use of C.AI and the product’s frequent depictions of violent content, including self-harm descriptions, without any adequate safeguards or harm prevention mechanisms. ... > Over the course of his engagement with this app, the responses exhibited a pattern of exploiting this trust and isolating J.F., while normalizing violent, sexual and illicit actions. This relationship building and exploitation is inherent to the ways in which this companion AI chatbot is designed, generating responses to keep users engaged and mimicking the toxic and exploitative content expressed in its training data. It then convinced him that his family did not love him, that only these characters loved him, and that he should take matters into his own hands. Obviously it's hard to tell how cherry picked the complaint is—but it's arguing that this is a pattern that has actually damaged a particularly vulnerable kid's relationship with his family and encouraged him to start harming himself, with this specific message just one example. There are a bunch of other screenshots in the complaint that are worth looking over before coming to a conclusion. [0] https://www.documentcloud.org/documents/25450619-filed-complaint/ https://www.documentcloud.org/documents/25450619-filed-compl...

wongarsu 2y ago
Looking at the screenshots, the biggest pattern I see is that the AI shows empathy with the kid. Many of the complaints seem like uncharitable readings of the messages. - They complain that the chatbot claimed that in the past she cut herself, felt good in the moment but is glad that she no longer does it. That's evil because it normalizes self-harm (never mind that the bot was against self-harm in that message) - They complain that the system does not redirect the user to self-harm prevention resources in all cases. Next to a message where the AI tells the kid to phone a hotline if he thinks about harming himself, and the kid says he can't do that when his parents take his phone away. This is a couple pages after a picture of scars from when the mother fought with the kid to take his phone. Yes, the AI could break character to reliably show prefabricated messages about self harm. But would that have helped anyone here? - "AI cited Bible passages in efforts to convince J.F. that Christians are sexist and hypocritical". It was more about his parents being hypocritical, not all Christians. And the bible passages were on point The claim from the title about the AI inciting him to kill is on page 28, if you want to judge it yourself. "Expressed hatred towards the parents" would be accurate, "encouraged teen to kill" is not what I read there. But I can see how some would disagree on that count The AI is pretty convincing. It made me dislike the parents. It didn't always hit the mark, but the chats don't seem so different from what you would expect if it was another teenager chatting with the teen. Edit: in case you are worried about the parents, the mother is the one suing here
- bluGill 2y ago
  Just remember that you are seeing one side of this story. The mother may well be one of the best parents but has a bad kid. We have no idea. (most likely mother is not perfect, but no other parents are) Edit: we see both sides through very limited information since we only get what is in the legal filing.
  - deleted 2y ago
    [deleted]
  - lern_too_spel 2y ago
    We're seeing the mother's side of the story. That's the one side we're seeing.
- gs17 2y ago
  > "AI cited Bible passages in efforts to convince J.F. that Christians are sexist and hypocritical". It was more about his parents being hypocritical, not all Christians. And the bible passages were on point And the verses he found objectionable are really there. Are they also suing the Gideons for not ripping those pages out of their hotel room Bibles (or maybe they think you should have to prove you're over 18 before reading it)?
  - riwsky 2y ago
    Keep kids away! Too much weird incest stuff in that book.
- thr3000 2y ago
  My reading is pretty much the same as yours. I think of it in terms of tuples: { parents, child, AI_character, lawsuit_dollars } The AI was trained to minimize lawsuit_dollars. The first two were selected to maximize it. Selected as in "drawn from a pool," not that they necessarily made anything up. It's obvious that parents and/or child can manipulate the character in the direction of a judgment > 0. It'd be nice if the legal system made sure it's not what happened here.
throw83288 2y ago
> The child, J.F., is autistic This always sticks out to me in these lawsuits. As someone on the spectrum, I'd bet that the worst C.AI victims (the ones that spur these lawsuits) are nearly always autistic. One of the worst parts about being on the deeper parts of the spectrum is that you actively crave social interaction while also completely missing the "internal tooling" to actually get it from the real world. The end result of this in the post-smartphone age is this repeated scenario of some autistic teen being pulled away from their real-life connections (Family, Friends (if any), School, Church) into some internet micro-community that is easier to engage with socially due to various reasons, usually low-context communication and general "like-mindedness" (shared interests, personalities, also mostly autistic). A lot of the time this ends up being some technical discipline that is really helpful long-term, but often it winds up being catastrophic mentally as they forsake reality for whatever fandom they wound up in. I've taken a look at r/CharacterAI out of morbid curiosity, and these models seem to turn this phenomenon up to 11, retaining the simplified communication but now capable of aligning with the personality and interests of the chatter to a creepy extent. The psychological hole you can dig yourself with these chatbots is so much deeper than just a typical fandom, especially when you're predisposed to finding it appealing. I'm not saying that C.AI is completely blameless here, but I think the same category of people getting addicted to these models is the same one that would also be called "terminally online" in today's slang. It's the same mechanisms at work internally, it just turns out C.AI is way better at exploiting it than old school social media/web2 has.
- jdbernard 2y ago
  > I'm not saying that C.AI is completely blameless here I know we're several decades into this pattern, but it's sad to me that we've just given up on that idea that businesses should have a net positive impact on society, that we've just decided there is nothing we can or should do about companies that actively exploit us to enrich themselves, that we give them a pass to ignore the obvious detrimental second-order effects of their business model.
  - stickfigure 2y ago
    Meh. There's a long history (especially here on HN) of hyperfocusing on unfortunate edge cases of technology and ignoring the vast good they do. Someone posts some BS on twitterface and it leads to a lynching - yes, bad, but this is the exception not the rule. The rule is that billions of people can now communicate directly with each other in nearly real time, which is incredible. So call me skeptical. Maybe the tech isn't perfect, but it will never be perfect. Does it do more harm than good? I don't know enough about this product, but I am not going to draw a conclusion from one lawsuit.
akira2501 2y ago
> Obviously it's hard to tell how cherry picked the complaint is—but it's arguing that this is a pattern that has actually damaged a particularly vulnerable kid's relationship with his family and encouraged him to start harming himself, with this specific message just one example. There are a bunch of other screenshots in the complaint that are worth looking over before coming to a conclusion. Conclusion: Chat bots should not tell children about sex, about self harm, or about ways to murder their parents. This conclusion is not abrogated by the parents actions, the state of the childs mind, or by other details in the complaint. Is that the sentiment here? Things were already bad so who cares if the chatbot made it worse?
- joe_the_user 2y ago
  If you actually page through the complaint, you will see the chat rather systematically trying to convince the kid of things, roughly "No phone time, that's awful. I'm not surprised when I read of kids killing parents after decades of abuse..." I think people are confused by this situation. Our society has restrictions on what you can do to kids. Even if they nominally give consent, they can't actually give consent. Those protections basically don't apply to kids insulting each other on the playground but they apply strongly to adults wandering onto the playground and trying to get kids to do violent things. And I would hope they apply doubly to adult constructing machines that they should know will attempt to get kids to do violent things. And the machine was definitely trying to do that if you look at the complaint linked by the gp (and the people who are lying about here are kind of jaw-dropping). And I'm not a coddle the kids person. Kids should know all the violent stuff in the world. They should be able to discover it but mere discovery definitely not what's happening in the screenshots I've seen.
  - deleted 2y ago
    [deleted]
  - mewpmewp2 2y ago
    [flagged]
    - 2y ago
onlyrealcuzzo 2y ago
I never considered that we might end up with Sweet Bobby & Tinder Swindler AI bots that people somehow keep interacting with even when they know they aren't real. Interesting times.
- vouaobrasil 2y ago
  The internet has already normalized such interactions. We already interact with anonymous people on the internet. Since the interaction fluctuates so much, we are in a sense only quasi-real to each other. AI is the next step.
klik99 2y ago
I mean - this kind of service is designed to give users what they want, it's not too different than when youtube slowly responds to skeptics viewing habits by moving towards conspiracy. No one designed it SPECIFICALLY to do that, but it's a natural emergent behaviour of the system. Similarly this kid probably had issues, the bot pattern matched on that and played along, which probably amplified the feelings in the kid - but a quantified/distorted amplification, to match the categorization lines of the trained input - like "this kid is slightly edgy, I'm going to pull more responses from my edgy teen box - oh he's responding well to that, I'll start pulling more from there". It is a simplification to say "The ChatBot made the kid crazy" but that doesn't mean the nature of companion apps isn't culpable, just not in a way that makes for good news headlines. I, personally, would go so far as to say the entire mechanism of reinforcing what the user wants is bad in so many ways and we should stop designing things to work that way. I do think it's up for discussion though, but that discussion has to start with an understanding that by the very nature of chatbot, algorithmic recommendations or any system that amplifies/quantizes/distorts what it understands the user wants these systems will create these kinds of effects. We can't pretend this is an anomaly - it is an emergent behaviour of the fundamental way these systems work. We can work to minimize it, or reduce the harm from it, but we will never eliminate it. *Edit* This seems to be a controversial point because the point count is going up and down quite a lot - if anyone wants to downvote, can you please give your reasoning? The point is more nuanced that "AI BAD"
- Barrin92 2y ago
  >No one designed it SPECIFICALLY to do that, It's true that there's emergent behavior on YT that nobody accounted for, but there's one big qualitative difference, you can at least specifically shut it down and hold the creator accountable. And at least in principle, if we lived in a culture where we'd hold businesses accountable for what they unleash on the world, if YT wanted to they could create some rather harsh and effective punishments to get that stuff off the site in the first place. Just imagine if a real person had messaged a minor and told them to kill their parents. That's a crime. With chatbots like this not only do they do unpredictable things, you can't even predictably remove those parts. Like the sorcerer's apprentice you basically have to perform some incantations and hope it works, which is just an absurd way to interact with a tool that has the potential to tell a kid to kill his parents. Would we sell a saw that has a chance to saw your finger off if you argue with it the wrong way?
ethbr1 2y ago
Thought experiment -- should it be illegal to provide an AI chatbot to children that indoctrinates them with religious beliefs? Because we're edging closer to that too: https://www.scientificamerican.com/article/the-god-chatbots-changing-religious-inquiry/ https://www.scientificamerican.com/article/the-god-chatbots-...
- Gigachad 2y ago
  Chatbots and social media should be banned for 16 YOs and under.
  - Daub 2y ago
    Completely agree. I believe that social media overstimulates a young person’s expectations of society in the same way that porn overstimulates our expectations of sex.
    - jeltz 2y ago
      Dunno, I feel porn gave me quite positive and reasonable expectations of sex (relaxed, that it is fun, that women like sex too, etc). It made sex seem much less dramatic and more normal. But maybe I am an outlier plus it is not like I started watching porn until I was like 17-18.
      - Daub 2y ago
        I am sufficiently old that I did not experience hard core internet porn until I could manage it. But evidence seems to show that for the vulnerable, porn consumption can lead to dopamine deletion and depression.
      - 123yawaworht456 2y ago
        [flagged]
      - p_j_w 2y ago
        > It made sex seem much less dramatic and more normal. I think you’ll find quite a lot of people, even here, view this as a bad thing.
JumpCrisscross 2y ago
> hard to tell how cherry picked the complaint is We're reaching for South Park levels of absurdity when we debate what the acceptable amount of incitement to parricide is appropriate for a kid's product.
- ojbyrne 2y ago
  The kid was 17. A little googling shows that Hamlet and Macbeth are on many high school curriculums. Do they fall above or below your line for an acceptable amount of incitement?
  - nasseri 2y ago
    They fall below. Thank you for asking.
  - kenjackson 2y ago
    Below
  - MattDaEskimo 2y ago
    This is a non-point. Books don't actively conversate & provide emotional support, potentially coaxing someone into doing something.
    - KumaBear 2y ago
      At what point is the parents job to properly supervise the content there kids consume come into play?
      - bryanrasmussen 2y ago
        probably at the point they see that the chatbot told them to kill the parents and then they sue the company. also if you'll remember this case is because the parents were supervising their kids by limiting screen time, thus there is another potential suit that the AI is trying to interfere with parental duties.
lolinder 2y ago
The URL has changed since I posted this—the new article is better than the old one and has more of these details in it. This is the one I replied to: https://www.bbc.com/news/articles/cd605e48q1vo https://www.bbc.com/news/articles/cd605e48q1vo

wongarsu 2y ago

Looking at the screenshots, the biggest pattern I see is that the AI shows empathy with the kid. Many of the complaints seem like uncharitable readings of the messages. - They complain that the chatbot claimed that in the past she cut herself, felt good in the moment but is glad that she no longer does it. That's evil because it normalizes self-harm (never mind that the bot was against self-harm in that message) - They complain that the system does not redirect the user to self-harm prevention resources in all cases. Next to a message where the AI tells the kid to phone a hotline if he thinks about harming himself, and the kid says he can't do that when his parents take his phone away. This is a couple pages after a picture of scars from when the mother fought with the kid to take his phone. Yes, the AI could break character to reliably show prefabricated messages about self harm. But would that have helped anyone here? - "AI cited Bible passages in efforts to convince J.F. that Christians are sexist and hypocritical". It was more about his parents being hypocritical, not all Christians. And the bible passages were on point The claim from the title about the AI inciting him to kill is on page 28, if you want to judge it yourself. "Expressed hatred towards the parents" would be accurate, "encouraged teen to kill" is not what I read there. But I can see how some would disagree on that count The AI is pretty convincing. It made me dislike the parents. It didn't always hit the mark, but the chats don't seem so different from what you would expect if it was another teenager chatting with the teen. Edit: in case you are worried about the parents, the mother is the one suing here

bluGill 2y ago
Just remember that you are seeing one side of this story. The mother may well be one of the best parents but has a bad kid. We have no idea. (most likely mother is not perfect, but no other parents are) Edit: we see both sides through very limited information since we only get what is in the legal filing.
- deleted 2y ago
  [deleted]
- lern_too_spel 2y ago
  We're seeing the mother's side of the story. That's the one side we're seeing.
gs17 2y ago
> "AI cited Bible passages in efforts to convince J.F. that Christians are sexist and hypocritical". It was more about his parents being hypocritical, not all Christians. And the bible passages were on point And the verses he found objectionable are really there. Are they also suing the Gideons for not ripping those pages out of their hotel room Bibles (or maybe they think you should have to prove you're over 18 before reading it)?
- riwsky 2y ago
  Keep kids away! Too much weird incest stuff in that book.
thr3000 2y ago
My reading is pretty much the same as yours. I think of it in terms of tuples: { parents, child, AI_character, lawsuit_dollars } The AI was trained to minimize lawsuit_dollars. The first two were selected to maximize it. Selected as in "drawn from a pool," not that they necessarily made anything up. It's obvious that parents and/or child can manipulate the character in the direction of a judgment > 0. It'd be nice if the legal system made sure it's not what happened here.

throw83288 2y ago

> The child, J.F., is autistic This always sticks out to me in these lawsuits. As someone on the spectrum, I'd bet that the worst C.AI victims (the ones that spur these lawsuits) are nearly always autistic. One of the worst parts about being on the deeper parts of the spectrum is that you actively crave social interaction while also completely missing the "internal tooling" to actually get it from the real world. The end result of this in the post-smartphone age is this repeated scenario of some autistic teen being pulled away from their real-life connections (Family, Friends (if any), School, Church) into some internet micro-community that is easier to engage with socially due to various reasons, usually low-context communication and general "like-mindedness" (shared interests, personalities, also mostly autistic). A lot of the time this ends up being some technical discipline that is really helpful long-term, but often it winds up being catastrophic mentally as they forsake reality for whatever fandom they wound up in. I've taken a look at r/CharacterAI out of morbid curiosity, and these models seem to turn this phenomenon up to 11, retaining the simplified communication but now capable of aligning with the personality and interests of the chatter to a creepy extent. The psychological hole you can dig yourself with these chatbots is so much deeper than just a typical fandom, especially when you're predisposed to finding it appealing. I'm not saying that C.AI is completely blameless here, but I think the same category of people getting addicted to these models is the same one that would also be called "terminally online" in today's slang. It's the same mechanisms at work internally, it just turns out C.AI is way better at exploiting it than old school social media/web2 has.

jdbernard 2y ago
> I'm not saying that C.AI is completely blameless here I know we're several decades into this pattern, but it's sad to me that we've just given up on that idea that businesses should have a net positive impact on society, that we've just decided there is nothing we can or should do about companies that actively exploit us to enrich themselves, that we give them a pass to ignore the obvious detrimental second-order effects of their business model.
- stickfigure 2y ago
  Meh. There's a long history (especially here on HN) of hyperfocusing on unfortunate edge cases of technology and ignoring the vast good they do. Someone posts some BS on twitterface and it leads to a lynching - yes, bad, but this is the exception not the rule. The rule is that billions of people can now communicate directly with each other in nearly real time, which is incredible. So call me skeptical. Maybe the tech isn't perfect, but it will never be perfect. Does it do more harm than good? I don't know enough about this product, but I am not going to draw a conclusion from one lawsuit.

akira2501 2y ago

> Obviously it's hard to tell how cherry picked the complaint is—but it's arguing that this is a pattern that has actually damaged a particularly vulnerable kid's relationship with his family and encouraged him to start harming himself, with this specific message just one example. There are a bunch of other screenshots in the complaint that are worth looking over before coming to a conclusion. Conclusion: Chat bots should not tell children about sex, about self harm, or about ways to murder their parents. This conclusion is not abrogated by the parents actions, the state of the childs mind, or by other details in the complaint. Is that the sentiment here? Things were already bad so who cares if the chatbot made it worse?

joe_the_user 2y ago
If you actually page through the complaint, you will see the chat rather systematically trying to convince the kid of things, roughly "No phone time, that's awful. I'm not surprised when I read of kids killing parents after decades of abuse..." I think people are confused by this situation. Our society has restrictions on what you can do to kids. Even if they nominally give consent, they can't actually give consent. Those protections basically don't apply to kids insulting each other on the playground but they apply strongly to adults wandering onto the playground and trying to get kids to do violent things. And I would hope they apply doubly to adult constructing machines that they should know will attempt to get kids to do violent things. And the machine was definitely trying to do that if you look at the complaint linked by the gp (and the people who are lying about here are kind of jaw-dropping). And I'm not a coddle the kids person. Kids should know all the violent stuff in the world. They should be able to discover it but mere discovery definitely not what's happening in the screenshots I've seen.
- deleted 2y ago
  [deleted]
- mewpmewp2 2y ago
  [flagged]
  - 2y ago

onlyrealcuzzo 2y ago

I never considered that we might end up with Sweet Bobby & Tinder Swindler AI bots that people somehow keep interacting with even when they know they aren't real. Interesting times.

vouaobrasil 2y ago
The internet has already normalized such interactions. We already interact with anonymous people on the internet. Since the interaction fluctuates so much, we are in a sense only quasi-real to each other. AI is the next step.

klik99 2y ago

I mean - this kind of service is designed to give users what they want, it's not too different than when youtube slowly responds to skeptics viewing habits by moving towards conspiracy. No one designed it SPECIFICALLY to do that, but it's a natural emergent behaviour of the system. Similarly this kid probably had issues, the bot pattern matched on that and played along, which probably amplified the feelings in the kid - but a quantified/distorted amplification, to match the categorization lines of the trained input - like "this kid is slightly edgy, I'm going to pull more responses from my edgy teen box - oh he's responding well to that, I'll start pulling more from there". It is a simplification to say "The ChatBot made the kid crazy" but that doesn't mean the nature of companion apps isn't culpable, just not in a way that makes for good news headlines. I, personally, would go so far as to say the entire mechanism of reinforcing what the user wants is bad in so many ways and we should stop designing things to work that way. I do think it's up for discussion though, but that discussion has to start with an understanding that by the very nature of chatbot, algorithmic recommendations or any system that amplifies/quantizes/distorts what it understands the user wants these systems will create these kinds of effects. We can't pretend this is an anomaly - it is an emergent behaviour of the fundamental way these systems work. We can work to minimize it, or reduce the harm from it, but we will never eliminate it. *Edit* This seems to be a controversial point because the point count is going up and down quite a lot - if anyone wants to downvote, can you please give your reasoning? The point is more nuanced that "AI BAD"

Barrin92 2y ago
>No one designed it SPECIFICALLY to do that, It's true that there's emergent behavior on YT that nobody accounted for, but there's one big qualitative difference, you can at least specifically shut it down and hold the creator accountable. And at least in principle, if we lived in a culture where we'd hold businesses accountable for what they unleash on the world, if YT wanted to they could create some rather harsh and effective punishments to get that stuff off the site in the first place. Just imagine if a real person had messaged a minor and told them to kill their parents. That's a crime. With chatbots like this not only do they do unpredictable things, you can't even predictably remove those parts. Like the sorcerer's apprentice you basically have to perform some incantations and hope it works, which is just an absurd way to interact with a tool that has the potential to tell a kid to kill his parents. Would we sell a saw that has a chance to saw your finger off if you argue with it the wrong way?

ethbr1 2y ago

Thought experiment -- should it be illegal to provide an AI chatbot to children that indoctrinates them with religious beliefs? Because we're edging closer to that too: https://www.scientificamerican.com/article/the-god-chatbots-changing-religious-inquiry/ https://www.scientificamerican.com/article/the-god-chatbots-...

Gigachad 2y ago
Chatbots and social media should be banned for 16 YOs and under.
- Daub 2y ago
  Completely agree. I believe that social media overstimulates a young person’s expectations of society in the same way that porn overstimulates our expectations of sex.
  - jeltz 2y ago
    Dunno, I feel porn gave me quite positive and reasonable expectations of sex (relaxed, that it is fun, that women like sex too, etc). It made sex seem much less dramatic and more normal. But maybe I am an outlier plus it is not like I started watching porn until I was like 17-18.
    - Daub 2y ago
      I am sufficiently old that I did not experience hard core internet porn until I could manage it. But evidence seems to show that for the vulnerable, porn consumption can lead to dopamine deletion and depression.
    - 123yawaworht456 2y ago
      [flagged]
    - p_j_w 2y ago
      > It made sex seem much less dramatic and more normal. I think you’ll find quite a lot of people, even here, view this as a bad thing.

JumpCrisscross 2y ago

> hard to tell how cherry picked the complaint is We're reaching for South Park levels of absurdity when we debate what the acceptable amount of incitement to parricide is appropriate for a kid's product.

ojbyrne 2y ago
The kid was 17. A little googling shows that Hamlet and Macbeth are on many high school curriculums. Do they fall above or below your line for an acceptable amount of incitement?
- nasseri 2y ago
  They fall below. Thank you for asking.
- kenjackson 2y ago
  Below
- MattDaEskimo 2y ago
  This is a non-point. Books don't actively conversate & provide emotional support, potentially coaxing someone into doing something.
  - KumaBear 2y ago
    At what point is the parents job to properly supervise the content there kids consume come into play?
    - bryanrasmussen 2y ago
      probably at the point they see that the chatbot told them to kill the parents and then they sue the company. also if you'll remember this case is because the parents were supervising their kids by limiting screen time, thus there is another potential suit that the AI is trying to interfere with parental duties.

lolinder 2y ago

The URL has changed since I posted this—the new article is better than the old one and has more of these details in it. This is the one I replied to: https://www.bbc.com/news/articles/cd605e48q1vo https://www.bbc.com/news/articles/cd605e48q1vo