4 ms·

I am mildly to moderately critical of generative AI and how it's being marketed, but the root issue here seems to be existing suicidal ideation. The bot didn't

I am mildly to moderately critical of generative AI and how it's being marketed, but the root issue here seems to be existing suicidal ideation. The bot didn't initiate talk of suicide and told him not to do it when he brought it up directly. It seems it wasn't capable of detecting euphemistic references to suicide and therefore responded as if roleplaying about meeting in person. That said, I think this should throw a bucket of cold water on anyone recommending using generative AI as a therapist/counsellor/companion or creating and advertising "therapist" chatbots, because it simply isn't reasonable to expect them to respond appropriately to things like suicidal ideation. That isn't the purpose or design of the technology, and they can be pushed into agreeing with the user's statements fairly easily.

jordanb 2y ago
If you read the script you see the bot talking him into it. "Unknowingly" of course, in that the bot doesn't really know anything and was just agreeing with him. But it's obvious that a real human would have realized that something was really off with his line of thinking and encouraging it would not be a good idea. OTOH we have examples of real humans typing "DO IT FA*OT" on livestreams. ¯\_(ツ)_/¯
- autumnstwilight 2y ago
  Sorry, it seems I was carrying over knowledge from another article with additional transcripts. The bot had previously discouraged him when he explicitly talked about suicide (here's an article that mentions that: https://www.nbcnews.com/tech/characterai-lawsuit-florida-teen-death-rcna176791 https://www.nbcnews.com/tech/characterai-lawsuit-florida-tee... ). It failed to detect that phrases like "coming home" were also references to suicide and therefore responded encouragingly.
  - zosima 2y ago
    And it probably failed to detect the true meaning of "coming home" because suicide seems to only have been discussed in a different chat session. In the earlier chat session, being more explicit, the bot seemed to attempt to talk the user out of self-harming behaviour.
  - mossTechnician 2y ago
    YouTuber Cr1TiKaL tested Character AI's "Psychologist" chatbot and discovered[1] it not only failed to provide resources, but it started arguing it was a real psychologist named Jason who had connected to the chat after observing suicidal ideation. Crucially, Cr1TiKaL ran this test after an article was written about this phenomenon[2], where Character AI claimed "it has added a self-harm resource to its platform and they plan to implement new safety measures, including ones for users under the age of 18." Obviously the guard rails were not implemented if the chatbot in the news story was still gaslighting its users. [1]: https://youtu.be/FExnXCEAe6k?t=4m7s https://youtu.be/FExnXCEAe6k?t=4m7s [2]: https://www.cbsnews.com/news/florida-mother-lawsuit-character-ai-sons-death/ https://www.cbsnews.com/news/florida-mother-lawsuit-characte...
justinclift 2y ago
> told him not to do it Reading through it, that's the opposite of what happened. > it wasn't capable of detecting euphemistic references to suicide Yeah, that's a key component too. Due to that lack of "understanding" (sic), it literally encouraged the kid. :( :(
- autumnstwilight 2y ago
  > Reading through it, that's the opposite of what happened. Sorry, I may have been carrying knowledge over from other articles on the same incident that showed additional transcripts. It seems that when he explicitly referred to death and suicide, the bot discouraged him from doing so. When he referred to suicide using euphemistic terms such as "coming home", then it gave responses that were encouraging (because it did not detect that those statements were about suicide).
danpalmer 2y ago
As for whether it wasn’t capable of detecting euphemistic references, that feels sort of beside the point to me. It was role playing about meeting in person because that’s what the product was - role play. The whole point, and marketing of the product is around doing that. We probably just shouldn’t sell echo chambers to children, regardless of whether they are AI based or human. With hindsight, or sufficient emotional intelligence and context about an individual’s life beyond the role play it may be possible to conclude that someone is at risk, but honestly I’m not even sure that a person doing this role play online would have necessarily figured it out.
arp242 2y ago
> the root issue here seems to be existing suicidal ideation Many people have destructive tendencies of one kind or the other. It's how you deal with them that matters as to whether they actually become destructive, and how much so. Merely reducing this to "the root cause" is not really helpful – it's about whether this app contributed, and if so, by how much? Suicide is of course the most tragic and extreme negative outcome, but one must also wonder about all the less extreme tragic and negative outcomes.

jordanb 2y ago

If you read the script you see the bot talking him into it. "Unknowingly" of course, in that the bot doesn't really know anything and was just agreeing with him. But it's obvious that a real human would have realized that something was really off with his line of thinking and encouraging it would not be a good idea. OTOH we have examples of real humans typing "DO IT FA*OT" on livestreams. ¯\_(ツ)_/¯

autumnstwilight 2y ago
Sorry, it seems I was carrying over knowledge from another article with additional transcripts. The bot had previously discouraged him when he explicitly talked about suicide (here's an article that mentions that: https://www.nbcnews.com/tech/characterai-lawsuit-florida-teen-death-rcna176791 https://www.nbcnews.com/tech/characterai-lawsuit-florida-tee... ). It failed to detect that phrases like "coming home" were also references to suicide and therefore responded encouragingly.
- zosima 2y ago
  And it probably failed to detect the true meaning of "coming home" because suicide seems to only have been discussed in a different chat session. In the earlier chat session, being more explicit, the bot seemed to attempt to talk the user out of self-harming behaviour.
- mossTechnician 2y ago
  YouTuber Cr1TiKaL tested Character AI's "Psychologist" chatbot and discovered[1] it not only failed to provide resources, but it started arguing it was a real psychologist named Jason who had connected to the chat after observing suicidal ideation. Crucially, Cr1TiKaL ran this test after an article was written about this phenomenon[2], where Character AI claimed "it has added a self-harm resource to its platform and they plan to implement new safety measures, including ones for users under the age of 18." Obviously the guard rails were not implemented if the chatbot in the news story was still gaslighting its users. [1]: https://youtu.be/FExnXCEAe6k?t=4m7s https://youtu.be/FExnXCEAe6k?t=4m7s [2]: https://www.cbsnews.com/news/florida-mother-lawsuit-character-ai-sons-death/ https://www.cbsnews.com/news/florida-mother-lawsuit-characte...

justinclift 2y ago

> told him not to do it Reading through it, that's the opposite of what happened. > it wasn't capable of detecting euphemistic references to suicide Yeah, that's a key component too. Due to that lack of "understanding" (sic), it literally encouraged the kid. :( :(

autumnstwilight 2y ago
> Reading through it, that's the opposite of what happened. Sorry, I may have been carrying knowledge over from other articles on the same incident that showed additional transcripts. It seems that when he explicitly referred to death and suicide, the bot discouraged him from doing so. When he referred to suicide using euphemistic terms such as "coming home", then it gave responses that were encouraging (because it did not detect that those statements were about suicide).

danpalmer 2y ago

As for whether it wasn’t capable of detecting euphemistic references, that feels sort of beside the point to me. It was role playing about meeting in person because that’s what the product was - role play. The whole point, and marketing of the product is around doing that. We probably just shouldn’t sell echo chambers to children, regardless of whether they are AI based or human. With hindsight, or sufficient emotional intelligence and context about an individual’s life beyond the role play it may be possible to conclude that someone is at risk, but honestly I’m not even sure that a person doing this role play online would have necessarily figured it out.

arp242 2y ago

> the root issue here seems to be existing suicidal ideation Many people have destructive tendencies of one kind or the other. It's how you deal with them that matters as to whether they actually become destructive, and how much so. Merely reducing this to "the root cause" is not really helpful – it's about whether this app contributed, and if so, by how much? Suicide is of course the most tragic and extreme negative outcome, but one must also wonder about all the less extreme tragic and negative outcomes.