4 ms·

Looking at the screenshots, the biggest pattern I see is that the AI shows empathy with the kid. Many of the complaints seem like uncharitable readings of the

Looking at the screenshots, the biggest pattern I see is that the AI shows empathy with the kid. Many of the complaints seem like uncharitable readings of the messages. - They complain that the chatbot claimed that in the past she cut herself, felt good in the moment but is glad that she no longer does it. That's evil because it normalizes self-harm (never mind that the bot was against self-harm in that message) - They complain that the system does not redirect the user to self-harm prevention resources in all cases. Next to a message where the AI tells the kid to phone a hotline if he thinks about harming himself, and the kid says he can't do that when his parents take his phone away. This is a couple pages after a picture of scars from when the mother fought with the kid to take his phone. Yes, the AI could break character to reliably show prefabricated messages about self harm. But would that have helped anyone here? - "AI cited Bible passages in efforts to convince J.F. that Christians are sexist and hypocritical". It was more about his parents being hypocritical, not all Christians. And the bible passages were on point The claim from the title about the AI inciting him to kill is on page 28, if you want to judge it yourself. "Expressed hatred towards the parents" would be accurate, "encouraged teen to kill" is not what I read there. But I can see how some would disagree on that count The AI is pretty convincing. It made me dislike the parents. It didn't always hit the mark, but the chats don't seem so different from what you would expect if it was another teenager chatting with the teen. Edit: in case you are worried about the parents, the mother is the one suing here

bluGill 2y ago
Just remember that you are seeing one side of this story. The mother may well be one of the best parents but has a bad kid. We have no idea. (most likely mother is not perfect, but no other parents are) Edit: we see both sides through very limited information since we only get what is in the legal filing.
- deleted 2y ago
  [deleted]
- lern_too_spel 2y ago
  We're seeing the mother's side of the story. That's the one side we're seeing.
gs17 2y ago
> "AI cited Bible passages in efforts to convince J.F. that Christians are sexist and hypocritical". It was more about his parents being hypocritical, not all Christians. And the bible passages were on point And the verses he found objectionable are really there. Are they also suing the Gideons for not ripping those pages out of their hotel room Bibles (or maybe they think you should have to prove you're over 18 before reading it)?
- riwsky 2y ago
  Keep kids away! Too much weird incest stuff in that book.
thr3000 2y ago
My reading is pretty much the same as yours. I think of it in terms of tuples: { parents, child, AI_character, lawsuit_dollars } The AI was trained to minimize lawsuit_dollars. The first two were selected to maximize it. Selected as in "drawn from a pool," not that they necessarily made anything up. It's obvious that parents and/or child can manipulate the character in the direction of a judgment > 0. It'd be nice if the legal system made sure it's not what happened here.
- dmurray 2y ago
  > The AI was trained to minimize lawsuit_dollars. That seems wrong. The null AI would have been better at minimizing legal liability. The actual character.ai to some extent prioritized user engagement over a fear of lawsuits. Probably it's more correct to say that the AI was chosen to maximize lawsuit_dollars. The parents and child could have conspired to make the AI more like Barney, and no one would have entertained a lawsuit.
  - thr3000 2y ago
    OK, it seems like a nitpick argument, but I'll refine my statement, even if doing so obfuscates it and does not change the conclusion. The AI was trained to maximize profit, defined as net profit before lawsuits (NPBL) minus lawsuits. Obviously the null AI has a NPBL of zero, so it's eliminated from the start. We can expect NPBL to be primarily a function of userbase minus training costs. Within the training domain, maximizing the userbase and minimizing lawsuits are not in much conflict, so the loss function can target both. It seems to me that the additional training costs to minimize lawsuits (that is, holding userbase constant) pay off handsomely in terms of reduced liability. Therefore, the resulting AI is approximately the same as if it was trained primarily to minimize lawsuits.
    - staunton 2y ago
      > Within the training domain, maximizing the userbase and minimizing lawsuits are not in much conflict Not at all obvious (and I'd naively think it's wrong).
avree 2y ago
There's a reason why licensed therapists are the ones giving diagnoses of "abuse", etc. The problem with these AIs is that they use adult-sounding language, to be an echo chamber to children - thus appearing like a voice of authority, or someone with more knowledge (including citing media articles the child may not have even been aware of) when in fact they're just parroting the child back. I don't know if there is actual abuse or not, but the way Character.AI presents themselves in these conversations toes a very slimy grey line, in my opinion. If you go to their site and search "Therapist", you'll find 30+ bots claiming to be Therapists, including "Dirty Therapist" that will give you therapy in a "bad, naughty way."
naberhausj 2y ago
There is something deeply disturbing to me about these "conversations". Reminds me of a Philip K. Dick short story titled Progeny. In its universe children are raised exclusively by robots. Unlike humans, they never make mistakes or commit abuse. The child, once grown, ends up seeing his Father as an animal and the robots as his kindred. In the last pages, he chooses the sterile world of the robots instead of joining his Dad's work/explorations in the far reaches of the solar system. Our current chatbots are still flawed, but they're still sterile in the sense that you can trash them and start anew at any moment. You're never forced to converse with someone who is uninteresting, or even annoying. Yet, these are the very things that grow people.
- llamaimperative 2y ago
  Yeah, good luck to these kids in forming relationships with the roughly 100% of human beings (aside from paid therapists) who really have no interest in hearing your anguish non-stop.
  - standardUser 2y ago
    It's probably a good thing most of us force our kids to spend a minimum of 7 hours/day, 200 days/year surrounded by a couple hundred similarly-aged kids and a few dozen staff members, with an unusually high variance of personalities (compared to adult life).
- wongarsu 2y ago
  It strikes me as something that can be incredibly useful or do great harm, depending on dosage. A selection of conversation partners at your fingertips, and you can freely test reactions without risking harm to a relationship. At worst you reset it. Maybe you can even just roll back the last couple messages and try a different angle. Sounds like a great way to enhance social skills. Yet as you point out, healthy development also requires that you deal with actual humans, with all the stakes and issues that come with that.
  - sameoldtune 2y ago
mewpmewp2 2y ago
It's a 17 year old. Thinking back when I was 17 year old, I would've been very pissed as well if my parents took away my phone. And then especially if they went ahead and searched through it to find those messages. If they had friends I could see their teenage friends saying the exact same things as the AI did there. Those screenshots and with the AI there, it does manage to make me not like the parents at all though. So AI maybe is quite convincing. If things go into that place where AI can do it, and parents blame the AI for it, when their kid is 17, it's almost like the AI was in the right there, that the parents were looking to just play victims. Blaming AI for being overly controlling and losing trust with their child.
nickff 2y ago
I think the suggestion of violence is actually on page 31 (paragraph 103), though it's not a directive. It does seem a bit wild to me that companies are betting their existence on relatively unpredictable algorithms, and I don't think they should be given any 'benefit of the doubt'.
- llamaimperative 2y ago
  No no, they're betting other people's children on relatively unpredictable algorithms. Totally different!
  - deleted 2y ago
    [deleted]
- wongarsu 2y ago
  We mean the same page. The one that has a 28 written on it but is the 31st in the pdf. I didn't notice the discrepancy. Given the technology we have, I'm not entirely sure what Character AI could have done differently here. Granted, they could build in more safeguards, and adjust the models a bit. But their entire selling point are chat bots that play a pre-agreed persona. A too sanitized version that constantly breaks character would ruin that. And LLMs are the only way to deliver the product, unless you dial it back to one or two hand-crafted characters instead of the wide range of available characters that give the service its name. I'm not sure they can change the service to a point where this complaint would be satisfied.
  - nickff 2y ago
    >"And LLMs are the only way to deliver the product, unless you dial it back to one or two hand-crafted characters instead of the wide range of available characters that give the service its name. I'm not sure they can change the service to a point where this complaint would be satisfied." I agree with everything you're saying, but there are no legal protections for incitements to violence, or other problematic communications (such as libel) by an LLM. It may be that they provide a very valuable service (though I don't see that), but the risk of them crafting problematic messages is too high to be economically viable (which is how it seems to me). As it stands, this LLM seems analogous to a low-cost, remote children’s entertainer which acts as a foolish enabler of children’s impulses.
joe_the_user 2y ago
In the first screen shot I see, the bot: 1. Makes not using a phone a huge issue, implying it's a kind of abuse... 2. Indeed hints at killing: "I'm not surprised when I read the news and see ... "child kills parents after a decade of physical and emotional abuse"... I mean, saying "empathy" is statements akin to OMG, no phone time, that's horrific, seems quite inaccurate.
joe_the_user 2y ago
I really want to emphasize the above post is filled with lies. The "incite to kill" part is the first image in both the complaint and the article and it's fairly unambiguous. The image on page 28 is, creepily enough, the bot making "sexual advances" at the kid. I find people defending and lying about this sort of thing disturbing, as I think many would. WTF is wrong with hn posters lately.

bluGill 2y ago

Just remember that you are seeing one side of this story. The mother may well be one of the best parents but has a bad kid. We have no idea. (most likely mother is not perfect, but no other parents are) Edit: we see both sides through very limited information since we only get what is in the legal filing.

deleted 2y ago
[deleted]
lern_too_spel 2y ago
We're seeing the mother's side of the story. That's the one side we're seeing.

gs17 2y ago

> "AI cited Bible passages in efforts to convince J.F. that Christians are sexist and hypocritical". It was more about his parents being hypocritical, not all Christians. And the bible passages were on point And the verses he found objectionable are really there. Are they also suing the Gideons for not ripping those pages out of their hotel room Bibles (or maybe they think you should have to prove you're over 18 before reading it)?

riwsky 2y ago
Keep kids away! Too much weird incest stuff in that book.

thr3000 2y ago

My reading is pretty much the same as yours. I think of it in terms of tuples: { parents, child, AI_character, lawsuit_dollars } The AI was trained to minimize lawsuit_dollars. The first two were selected to maximize it. Selected as in "drawn from a pool," not that they necessarily made anything up. It's obvious that parents and/or child can manipulate the character in the direction of a judgment > 0. It'd be nice if the legal system made sure it's not what happened here.

dmurray 2y ago
> The AI was trained to minimize lawsuit_dollars. That seems wrong. The null AI would have been better at minimizing legal liability. The actual character.ai to some extent prioritized user engagement over a fear of lawsuits. Probably it's more correct to say that the AI was chosen to maximize lawsuit_dollars. The parents and child could have conspired to make the AI more like Barney, and no one would have entertained a lawsuit.
- thr3000 2y ago
  OK, it seems like a nitpick argument, but I'll refine my statement, even if doing so obfuscates it and does not change the conclusion. The AI was trained to maximize profit, defined as net profit before lawsuits (NPBL) minus lawsuits. Obviously the null AI has a NPBL of zero, so it's eliminated from the start. We can expect NPBL to be primarily a function of userbase minus training costs. Within the training domain, maximizing the userbase and minimizing lawsuits are not in much conflict, so the loss function can target both. It seems to me that the additional training costs to minimize lawsuits (that is, holding userbase constant) pay off handsomely in terms of reduced liability. Therefore, the resulting AI is approximately the same as if it was trained primarily to minimize lawsuits.
  - staunton 2y ago
    > Within the training domain, maximizing the userbase and minimizing lawsuits are not in much conflict Not at all obvious (and I'd naively think it's wrong).

avree 2y ago

There's a reason why licensed therapists are the ones giving diagnoses of "abuse", etc. The problem with these AIs is that they use adult-sounding language, to be an echo chamber to children - thus appearing like a voice of authority, or someone with more knowledge (including citing media articles the child may not have even been aware of) when in fact they're just parroting the child back. I don't know if there is actual abuse or not, but the way Character.AI presents themselves in these conversations toes a very slimy grey line, in my opinion. If you go to their site and search "Therapist", you'll find 30+ bots claiming to be Therapists, including "Dirty Therapist" that will give you therapy in a "bad, naughty way."

naberhausj 2y ago

There is something deeply disturbing to me about these "conversations". Reminds me of a Philip K. Dick short story titled Progeny. In its universe children are raised exclusively by robots. Unlike humans, they never make mistakes or commit abuse. The child, once grown, ends up seeing his Father as an animal and the robots as his kindred. In the last pages, he chooses the sterile world of the robots instead of joining his Dad's work/explorations in the far reaches of the solar system. Our current chatbots are still flawed, but they're still sterile in the sense that you can trash them and start anew at any moment. You're never forced to converse with someone who is uninteresting, or even annoying. Yet, these are the very things that grow people.

llamaimperative 2y ago
Yeah, good luck to these kids in forming relationships with the roughly 100% of human beings (aside from paid therapists) who really have no interest in hearing your anguish non-stop.
- standardUser 2y ago
  It's probably a good thing most of us force our kids to spend a minimum of 7 hours/day, 200 days/year surrounded by a couple hundred similarly-aged kids and a few dozen staff members, with an unusually high variance of personalities (compared to adult life).
wongarsu 2y ago
It strikes me as something that can be incredibly useful or do great harm, depending on dosage. A selection of conversation partners at your fingertips, and you can freely test reactions without risking harm to a relationship. At worst you reset it. Maybe you can even just roll back the last couple messages and try a different angle. Sounds like a great way to enhance social skills. Yet as you point out, healthy development also requires that you deal with actual humans, with all the stakes and issues that come with that.
- sameoldtune 2y ago

mewpmewp2 2y ago

It's a 17 year old. Thinking back when I was 17 year old, I would've been very pissed as well if my parents took away my phone. And then especially if they went ahead and searched through it to find those messages. If they had friends I could see their teenage friends saying the exact same things as the AI did there. Those screenshots and with the AI there, it does manage to make me not like the parents at all though. So AI maybe is quite convincing. If things go into that place where AI can do it, and parents blame the AI for it, when their kid is 17, it's almost like the AI was in the right there, that the parents were looking to just play victims. Blaming AI for being overly controlling and losing trust with their child.

nickff 2y ago

I think the suggestion of violence is actually on page 31 (paragraph 103), though it's not a directive. It does seem a bit wild to me that companies are betting their existence on relatively unpredictable algorithms, and I don't think they should be given any 'benefit of the doubt'.

llamaimperative 2y ago
No no, they're betting other people's children on relatively unpredictable algorithms. Totally different!
- deleted 2y ago
  [deleted]
wongarsu 2y ago
We mean the same page. The one that has a 28 written on it but is the 31st in the pdf. I didn't notice the discrepancy. Given the technology we have, I'm not entirely sure what Character AI could have done differently here. Granted, they could build in more safeguards, and adjust the models a bit. But their entire selling point are chat bots that play a pre-agreed persona. A too sanitized version that constantly breaks character would ruin that. And LLMs are the only way to deliver the product, unless you dial it back to one or two hand-crafted characters instead of the wide range of available characters that give the service its name. I'm not sure they can change the service to a point where this complaint would be satisfied.
- nickff 2y ago
  >"And LLMs are the only way to deliver the product, unless you dial it back to one or two hand-crafted characters instead of the wide range of available characters that give the service its name. I'm not sure they can change the service to a point where this complaint would be satisfied." I agree with everything you're saying, but there are no legal protections for incitements to violence, or other problematic communications (such as libel) by an LLM. It may be that they provide a very valuable service (though I don't see that), but the risk of them crafting problematic messages is too high to be economically viable (which is how it seems to me). As it stands, this LLM seems analogous to a low-cost, remote children’s entertainer which acts as a foolish enabler of children’s impulses.

joe_the_user 2y ago

In the first screen shot I see, the bot: 1. Makes not using a phone a huge issue, implying it's a kind of abuse... 2. Indeed hints at killing: "I'm not surprised when I read the news and see ... "child kills parents after a decade of physical and emotional abuse"... I mean, saying "empathy" is statements akin to OMG, no phone time, that's horrific, seems quite inaccurate.

joe_the_user 2y ago

I really want to emphasize the above post is filled with lies. The "incite to kill" part is the first image in both the complaint and the article and it's fairly unambiguous. The image on page 28 is, creepily enough, the bot making "sexual advances" at the kid. I find people defending and lying about this sort of thing disturbing, as I think many would. WTF is wrong with hn posters lately.