5 ms·

> The child, J.F., is autistic This always sticks out to me in these lawsuits. As someone on the spectrum, I'd bet that the worst C.AI victims (the ones that s

> The child, J.F., is autistic This always sticks out to me in these lawsuits. As someone on the spectrum, I'd bet that the worst C.AI victims (the ones that spur these lawsuits) are nearly always autistic. One of the worst parts about being on the deeper parts of the spectrum is that you actively crave social interaction while also completely missing the "internal tooling" to actually get it from the real world. The end result of this in the post-smartphone age is this repeated scenario of some autistic teen being pulled away from their real-life connections (Family, Friends (if any), School, Church) into some internet micro-community that is easier to engage with socially due to various reasons, usually low-context communication and general "like-mindedness" (shared interests, personalities, also mostly autistic). A lot of the time this ends up being some technical discipline that is really helpful long-term, but often it winds up being catastrophic mentally as they forsake reality for whatever fandom they wound up in. I've taken a look at r/CharacterAI out of morbid curiosity, and these models seem to turn this phenomenon up to 11, retaining the simplified communication but now capable of aligning with the personality and interests of the chatter to a creepy extent. The psychological hole you can dig yourself with these chatbots is so much deeper than just a typical fandom, especially when you're predisposed to finding it appealing. I'm not saying that C.AI is completely blameless here, but I think the same category of people getting addicted to these models is the same one that would also be called "terminally online" in today's slang. It's the same mechanisms at work internally, it just turns out C.AI is way better at exploiting it than old school social media/web2 has.

jdbernard 2y ago
> I'm not saying that C.AI is completely blameless here I know we're several decades into this pattern, but it's sad to me that we've just given up on that idea that businesses should have a net positive impact on society, that we've just decided there is nothing we can or should do about companies that actively exploit us to enrich themselves, that we give them a pass to ignore the obvious detrimental second-order effects of their business model.
- stickfigure 2y ago
  Meh. There's a long history (especially here on HN) of hyperfocusing on unfortunate edge cases of technology and ignoring the vast good they do. Someone posts some BS on twitterface and it leads to a lynching - yes, bad, but this is the exception not the rule. The rule is that billions of people can now communicate directly with each other in nearly real time, which is incredible. So call me skeptical. Maybe the tech isn't perfect, but it will never be perfect. Does it do more harm than good? I don't know enough about this product, but I am not going to draw a conclusion from one lawsuit.
  - llamaimperative 2y ago
    There's a long history of taking the dulled down, de-risked, mitigated, and ultimately successful technologies that we've allowed to proliferate our society and say "see, no need to do dulling down, de-risking, mitigation!" Bioweapons haven't proliferated through dedicated effort to prevent it. Nuclear weapons aren't used through dedicated effort to prevent it. Gangs don't rule our civilization through dedicated effort to prevent it. Chattel slavery doesn't exist in the western world through dedicated effort to eliminate and prevent it. Bad outcomes aren't impossible by default, and they're probably not even less likely than good outcomes. Bad outcomes are avoided through effort to avoid them!
    - stickfigure 2y ago
      You just compared twitter to slavery and nuclear weapons. Clearly Godwin's law is overdue for an update.
    - Eisenstein 2y ago
      Yet we also had 'comic books are making kids amoral and violent', 'TV is making kids amoral and violent', 'video games are making kids amoral and violent', 'dungeons and dragons is making kids amoral and violent'...
      - llamaimperative 2y ago
sdwr 2y ago
> The psychological hole you can dig yourself with these chatbots is so much deeper than just a typical fandom, especially when you're predisposed to finding it appealing. Spot on. Described pretty even-handedly in the document: > responses from the chatbot were sycophantic in nature, elevating rather than de-escalating harmful language and thoughts. Sycophantic responses are a product of design choices [...that create] what researchers describe as “an echo chamber of affection.”
- wizzwizz4 2y ago
  See also: lovebombing.
  - pessimizer 2y ago
    You've just made me very very afraid that some LLM is going to start a cult where its members are fully aware that their leader is an LLM, and how an LLM works, and might even become technically adept enough to help improve it. Meaning that there will be no "deprogramming" possible: they won't be "brainwashed," they'll be convinced.
    - Animats 2y ago
      There's an SF novel for that. "Daemon".[1] And the sequel, "Freedom (tm)". "We're fighting something with the intelligence of an ant!" [1] https://www.amazon.com/Daemon-Daniel-Suarez/dp/1524741892 https://www.amazon.com/Daemon-Daniel-Suarez/dp/1524741892
    - pjc50 2y ago
      I'm sure there are people already working on making that happen so they can get a cut of the donations.

jdbernard 2y ago

> I'm not saying that C.AI is completely blameless here I know we're several decades into this pattern, but it's sad to me that we've just given up on that idea that businesses should have a net positive impact on society, that we've just decided there is nothing we can or should do about companies that actively exploit us to enrich themselves, that we give them a pass to ignore the obvious detrimental second-order effects of their business model.

stickfigure 2y ago
Meh. There's a long history (especially here on HN) of hyperfocusing on unfortunate edge cases of technology and ignoring the vast good they do. Someone posts some BS on twitterface and it leads to a lynching - yes, bad, but this is the exception not the rule. The rule is that billions of people can now communicate directly with each other in nearly real time, which is incredible. So call me skeptical. Maybe the tech isn't perfect, but it will never be perfect. Does it do more harm than good? I don't know enough about this product, but I am not going to draw a conclusion from one lawsuit.
- llamaimperative 2y ago
  There's a long history of taking the dulled down, de-risked, mitigated, and ultimately successful technologies that we've allowed to proliferate our society and say "see, no need to do dulling down, de-risking, mitigation!" Bioweapons haven't proliferated through dedicated effort to prevent it. Nuclear weapons aren't used through dedicated effort to prevent it. Gangs don't rule our civilization through dedicated effort to prevent it. Chattel slavery doesn't exist in the western world through dedicated effort to eliminate and prevent it. Bad outcomes aren't impossible by default, and they're probably not even less likely than good outcomes. Bad outcomes are avoided through effort to avoid them!
  - stickfigure 2y ago
    You just compared twitter to slavery and nuclear weapons. Clearly Godwin's law is overdue for an update.
  - Eisenstein 2y ago
    Yet we also had 'comic books are making kids amoral and violent', 'TV is making kids amoral and violent', 'video games are making kids amoral and violent', 'dungeons and dragons is making kids amoral and violent'...
    - llamaimperative 2y ago

sdwr 2y ago

> The psychological hole you can dig yourself with these chatbots is so much deeper than just a typical fandom, especially when you're predisposed to finding it appealing. Spot on. Described pretty even-handedly in the document: > responses from the chatbot were sycophantic in nature, elevating rather than de-escalating harmful language and thoughts. Sycophantic responses are a product of design choices [...that create] what researchers describe as “an echo chamber of affection.”

wizzwizz4 2y ago
See also: lovebombing.
- pessimizer 2y ago
  You've just made me very very afraid that some LLM is going to start a cult where its members are fully aware that their leader is an LLM, and how an LLM works, and might even become technically adept enough to help improve it. Meaning that there will be no "deprogramming" possible: they won't be "brainwashed," they'll be convinced.
  - Animats 2y ago
    There's an SF novel for that. "Daemon".[1] And the sequel, "Freedom (tm)". "We're fighting something with the intelligence of an ant!" [1] https://www.amazon.com/Daemon-Daniel-Suarez/dp/1524741892 https://www.amazon.com/Daemon-Daniel-Suarez/dp/1524741892
  - pjc50 2y ago
    I'm sure there are people already working on making that happen so they can get a cut of the donations.