5 ms·

Chatbot Psychosis

sublinear 9mo ago
The interesting part to me is how this anti-feature could become the primary source of value for AI if only it was easier for everyone to run and train locally from a blank slate and without the clumsy natural language interface. Take the example of music. Most musicians probably don't want crap like Suno. What they actually want is a fountain of ideas to riff on based on a locally trained AI where they have finer-grained control over the parameters and attention. Instead of "telling" the AI "more this and less that" would it not make more sense to surface facets of the training data in a more transparent and comprehensible way, and provide a slider control or ability to completely eliminate certain influences? I'm aware that's probably a tall order, but it's what's necessary. Instead of producing delusions left to random chance and uncurated training data, we should be trying to guide AI towards clarity with the user in full control. The local training by the user effectively becomes a mirror of that user's artistic vision. It would be unique and not "owned" by anyone else.
- deleted 9mo ago
  [deleted]
derrida 9mo ago
As I understand it - a person with psychosis is someone who has over-weighted perceptions that cannot be corrected with sensory input. Hence to "bring someone to their senses". I've seen and thought there might be a few programmers maybe with a related (not psychosis) "ai mania" - where one thinks one changing the world and uniquely positioned. It's not that we're not capable in our small ways with network effects, or like a hand touch could begin the wave breaking (hello surfers!) what distinguishes this capacity for small effects having big impacts from the mania version is the mania version bears no subtle understanding of cause and effect. A person who is adept in understanding cause and effect usually there's quite a simple, beautiful and refined rule of thumb the know about it. "Less is more". Mania on the other hand proliferates outward - concept upon concept upon concept - and these concepts are removed from that cause and effect - playful interaction with nature. As a wise old sage hacker once said "Proof of concept, or get the fuck out" Where mania occurs in contrast to grounded capacity that does change the world is in the aspect of responsibility - being able to respond or be responsive to something. Someone who changes the world with their maths equation will be able to be responsive, responsible and claim, a manic person there's a disconnect. Actually, from certain points of view with certain apps or mediums that have a claim to universality and claiming "how the world is" it looks like there's most definitely already some titans of industry and powerful people inside the feedback loop of mania. The should just go for a walk. Call a childhood friend. Have a cup of tea. Come to their senses (literally). it's good to remember we're just querying a datastructure.
- strogonoff 9mo ago
  Humans don’t exist as standalone individuals, and a lot of our experience is shaped by being in a larger graph of people. I suspect it’s under-appreciated how social our perception is: in order to tell each other about X, we need to classify reality, define X and separate it from non-X; and because we (directly or indirectly) talk to each other so much, because we generally don’t just shut off the part of ourselves that classifies reality, the shared map we have for the purposes of communication blends with (perhaps even becomes) our experience of reality. So, to me, “to bring someone to their senses” is significantly about reinforcing a shared map through interpersonal connection—not unlike how before online forums it was much harder to maintain particularly unorthodox[0] worldviews: when exposure to a selection of people around you is non-optional, it tempers even the most extreme left-field takes, as humans (sans pathologies) are primed to mirror each other. I’m not a psychologist, but likening chatbot psychosis to an ungrounded feedback loop rings true, except I would say human connection is the missing ground (although you could say it’s not grounded in senses or experience by proxy, per above). Arguably, one of the significant issues of chatbots is the illusion of human connection where there’s nothing but a data structure query; and I know that some people have no trouble treating the chat as just that, but somehow that doesn’t seem like a consolation: if treating that which quite successfully pretends to be a natural conversation with a human as nothing more than a data structure query comes so naturally to them, what does it say about how they see conversing with us, the actual humans around them? [0] As in, starkly misaligned with the community—which admittedly could be for better or for worse (isolated cults come to mind).
  - andsoitis 9mo ago
    > if treating that which quite successfully pretends to be a natural conversation with a human as nothing more than a data structure query comes so naturally to them, what does it say about how they see conversing with us, the actual humans around them? You are suspicious of people who treat an AI chatbot for what it is, just a tool?
    - strogonoff 9mo ago
      “Suspicious” is your word. As the saying goes: if it fires together, it wires together. Is it outlandish to wonder whether, after you create a habit of using certain tricks (including lies, threats of physical violence[0], etc.) whenever your human-like counterpart doesn’t provide required output, you might use those with another human-like counterpart that just happens to also be an actual human? Whether one’s abuse of p-zombified perfect human copies might lead to a change in how one sees and treats actual humans, which are increasingly no different (if not worse) in their text output except they can also feel? I’m not a psychologist so I can’t say. Maybe some people have no issues treating this tool as a tool while their “system 2” is tirelessly making sure they are at all times mindful whether their fully human-like counterpart is or is not actually human. Maybe they actually see others around them as bots, except they suppress treating them like that out of fear of retribution. Who knows, maybe it’s not a pathology and we are all like that deep inside. Maybe this provides a vent for aggression and people who abuse chatbots might actually be nicer to other humans as a result. What we do know, though, is that the tool mimics human behaviour well enough that possibly even more other people (many presumably without diagnosed pathologies) treat it as very much human, some to the point of having [a]romantic relationships with it. [0] https://youtu.be/8g7a0IWKDRE?t=480 https://youtu.be/8g7a0IWKDRE?t=480
ramoz 9mo ago
This isn't limited to chat bots. There are clearly some developers experiencing it with coding agents. Maybe humans don't carry the cognitive capacity for so much, and so rapid, information-parsing; and we experience vulnerabilities (ie psychosis) with over consumption.
Aeglaecia 9mo ago
ten bucks says this condition ends up evolving in the same way that female hysteria did what if llms are actually equivalent to humans in sentience ? wouldn't that make everyone psychotic except those in "chatbot psychosis" ?
- Joel_Mckay 9mo ago
  Besides encouraging suicide, many LLM also feed peoples delusions =3 https://www.youtube.com/watch?v=yftBiNu0ZNU https://www.youtube.com/watch?v=yftBiNu0ZNU
- yoz-y 9mo ago
  Don’t know if it is the same for everyone. But when I experienced psychosis I definitely thought I was on a “higher plane” of thinking than others. That didn’t help me get a single idea through and of course it was all BS. So no, it definitely is not a desirable state of mind.
  - Aeglaecia 9mo ago
    i don't wish to debate the accuracy of your experience , however i will challenge it a bit - perhaps all psychotics believe they are on a higher plane of thinking , this does not imply that all those on a higher plane of thinking are psychotics
    - andsoitis 9mo ago
      What is this higher plane you’re talking about?
      - doublerabbit 9mo ago
        Astral, Spiritual plane. The same place that Psychonauts try to reach out too.
        immibis 9mo ago
Lerc 9mo ago
I would be interested to see a comparison between this and reefer madness. There was a time when media coverage could have easily convinced you of something that was not true. How can you distinguish reporting of a real phenomenon to that of a imagined one? Barring a scientific consensus being reached after significant analysis, do we just roll the dice and hope the guess that conforms to our prejudices turns out to be the right one.
FuturisticLover 9mo ago
I am shocked after finding out about so many deaths linked to chatbots - https://en.wikipedia.org/wiki/Deaths_linked_to_chatbots https://en.wikipedia.org/wiki/Deaths_linked_to_chatbots
- simulator5g 9mo ago
  Why would you be shocked? People talk about these things like they’re some kind of oracle. It’s pretty obvious that an “oracle” which gives somewhat random answers is dangerous.
  - JasonADrury 9mo ago
    The fact that a technology being used by literally billions of people has a wikipedia page listing only 15 related deaths does not seem to suggest that the technology is dangerous.
    - estearum 9mo ago
      I remember seeing people make similar arguments in the early days of OxyContin’s takeover. The length of the Wikipedia article at this point isn’t good evidence of either side.
    - simulator5g 9mo ago
      15 that we know of, so far. People outside the HN crowd just started using AI pretty recently. There’s also a case to be made that the increased pollution has a certain death toll as well. We know that fossil fuels, construction, manufacturing, and even noise all produce pollution that reduce the average lifespan in a given area. Some of these data centers are being run off gas turbines which are very dirty.
- JasonADrury 9mo ago
  With half the planet using these, that hardly seems like many.
lostmsu 9mo ago
> Journalistic accounts describe individuals who have developed strong beliefs that chatbots are sentient But chatbots are sentient within a single context session!
seamossfet 9mo ago
They missed an opportunity to call this "cyber psychosis"
Lapsa 9mo ago
given secret service is capable of reading thoughts - channeling delusions ain't that much of a far stretch
ottah 9mo ago
This isn't psychosis, this is a cult. It's new age religious beliefs, wrapped up in tech. These people aren't any different than people who believe in speaking in tongues, or transubstantiation.

The interesting part to me is how this anti-feature could become the primary source of value for AI if only it was easier for everyone to run and train locally from a blank slate and without the clumsy natural language interface. Take the example of music. Most musicians probably don't want crap like Suno. What they actually want is a fountain of ideas to riff on based on a locally trained AI where they have finer-grained control over the parameters and attention. Instead of "telling" the AI "more this and less that" would it not make more sense to surface facets of the training data in a more transparent and comprehensible way, and provide a slider control or ability to completely eliminate certain influences? I'm aware that's probably a tall order, but it's what's necessary. Instead of producing delusions left to random chance and uncurated training data, we should be trying to guide AI towards clarity with the user in full control. The local training by the user effectively becomes a mirror of that user's artistic vision. It would be unique and not "owned" by anyone else.

deleted 9mo ago
[deleted]

derrida 9mo ago

As I understand it - a person with psychosis is someone who has over-weighted perceptions that cannot be corrected with sensory input. Hence to "bring someone to their senses". I've seen and thought there might be a few programmers maybe with a related (not psychosis) "ai mania" - where one thinks one changing the world and uniquely positioned. It's not that we're not capable in our small ways with network effects, or like a hand touch could begin the wave breaking (hello surfers!) what distinguishes this capacity for small effects having big impacts from the mania version is the mania version bears no subtle understanding of cause and effect. A person who is adept in understanding cause and effect usually there's quite a simple, beautiful and refined rule of thumb the know about it. "Less is more". Mania on the other hand proliferates outward - concept upon concept upon concept - and these concepts are removed from that cause and effect - playful interaction with nature. As a wise old sage hacker once said "Proof of concept, or get the fuck out" Where mania occurs in contrast to grounded capacity that does change the world is in the aspect of responsibility - being able to respond or be responsive to something. Someone who changes the world with their maths equation will be able to be responsive, responsible and claim, a manic person there's a disconnect. Actually, from certain points of view with certain apps or mediums that have a claim to universality and claiming "how the world is" it looks like there's most definitely already some titans of industry and powerful people inside the feedback loop of mania. The should just go for a walk. Call a childhood friend. Have a cup of tea. Come to their senses (literally). it's good to remember we're just querying a datastructure.

strogonoff 9mo ago
Humans don’t exist as standalone individuals, and a lot of our experience is shaped by being in a larger graph of people. I suspect it’s under-appreciated how social our perception is: in order to tell each other about X, we need to classify reality, define X and separate it from non-X; and because we (directly or indirectly) talk to each other so much, because we generally don’t just shut off the part of ourselves that classifies reality, the shared map we have for the purposes of communication blends with (perhaps even becomes) our experience of reality. So, to me, “to bring someone to their senses” is significantly about reinforcing a shared map through interpersonal connection—not unlike how before online forums it was much harder to maintain particularly unorthodox[0] worldviews: when exposure to a selection of people around you is non-optional, it tempers even the most extreme left-field takes, as humans (sans pathologies) are primed to mirror each other. I’m not a psychologist, but likening chatbot psychosis to an ungrounded feedback loop rings true, except I would say human connection is the missing ground (although you could say it’s not grounded in senses or experience by proxy, per above). Arguably, one of the significant issues of chatbots is the illusion of human connection where there’s nothing but a data structure query; and I know that some people have no trouble treating the chat as just that, but somehow that doesn’t seem like a consolation: if treating that which quite successfully pretends to be a natural conversation with a human as nothing more than a data structure query comes so naturally to them, what does it say about how they see conversing with us, the actual humans around them? [0] As in, starkly misaligned with the community—which admittedly could be for better or for worse (isolated cults come to mind).
- andsoitis 9mo ago
  > if treating that which quite successfully pretends to be a natural conversation with a human as nothing more than a data structure query comes so naturally to them, what does it say about how they see conversing with us, the actual humans around them? You are suspicious of people who treat an AI chatbot for what it is, just a tool?
  - strogonoff 9mo ago
    “Suspicious” is your word. As the saying goes: if it fires together, it wires together. Is it outlandish to wonder whether, after you create a habit of using certain tricks (including lies, threats of physical violence[0], etc.) whenever your human-like counterpart doesn’t provide required output, you might use those with another human-like counterpart that just happens to also be an actual human? Whether one’s abuse of p-zombified perfect human copies might lead to a change in how one sees and treats actual humans, which are increasingly no different (if not worse) in their text output except they can also feel? I’m not a psychologist so I can’t say. Maybe some people have no issues treating this tool as a tool while their “system 2” is tirelessly making sure they are at all times mindful whether their fully human-like counterpart is or is not actually human. Maybe they actually see others around them as bots, except they suppress treating them like that out of fear of retribution. Who knows, maybe it’s not a pathology and we are all like that deep inside. Maybe this provides a vent for aggression and people who abuse chatbots might actually be nicer to other humans as a result. What we do know, though, is that the tool mimics human behaviour well enough that possibly even more other people (many presumably without diagnosed pathologies) treat it as very much human, some to the point of having [a]romantic relationships with it. [0] https://youtu.be/8g7a0IWKDRE?t=480 https://youtu.be/8g7a0IWKDRE?t=480

ramoz 9mo ago

This isn't limited to chat bots. There are clearly some developers experiencing it with coding agents. Maybe humans don't carry the cognitive capacity for so much, and so rapid, information-parsing; and we experience vulnerabilities (ie psychosis) with over consumption.

Aeglaecia 9mo ago

ten bucks says this condition ends up evolving in the same way that female hysteria did what if llms are actually equivalent to humans in sentience ? wouldn't that make everyone psychotic except those in "chatbot psychosis" ?

Joel_Mckay 9mo ago
Besides encouraging suicide, many LLM also feed peoples delusions =3 https://www.youtube.com/watch?v=yftBiNu0ZNU https://www.youtube.com/watch?v=yftBiNu0ZNU
yoz-y 9mo ago
Don’t know if it is the same for everyone. But when I experienced psychosis I definitely thought I was on a “higher plane” of thinking than others. That didn’t help me get a single idea through and of course it was all BS. So no, it definitely is not a desirable state of mind.
- Aeglaecia 9mo ago
  i don't wish to debate the accuracy of your experience , however i will challenge it a bit - perhaps all psychotics believe they are on a higher plane of thinking , this does not imply that all those on a higher plane of thinking are psychotics
  - andsoitis 9mo ago
    What is this higher plane you’re talking about?
    - doublerabbit 9mo ago
      Astral, Spiritual plane. The same place that Psychonauts try to reach out too.
      - immibis 9mo ago

Lerc 9mo ago

I would be interested to see a comparison between this and reefer madness. There was a time when media coverage could have easily convinced you of something that was not true. How can you distinguish reporting of a real phenomenon to that of a imagined one? Barring a scientific consensus being reached after significant analysis, do we just roll the dice and hope the guess that conforms to our prejudices turns out to be the right one.

FuturisticLover 9mo ago

I am shocked after finding out about so many deaths linked to chatbots - https://en.wikipedia.org/wiki/Deaths_linked_to_chatbots https://en.wikipedia.org/wiki/Deaths_linked_to_chatbots

simulator5g 9mo ago
Why would you be shocked? People talk about these things like they’re some kind of oracle. It’s pretty obvious that an “oracle” which gives somewhat random answers is dangerous.
- JasonADrury 9mo ago
  The fact that a technology being used by literally billions of people has a wikipedia page listing only 15 related deaths does not seem to suggest that the technology is dangerous.
  - estearum 9mo ago
    I remember seeing people make similar arguments in the early days of OxyContin’s takeover. The length of the Wikipedia article at this point isn’t good evidence of either side.
  - simulator5g 9mo ago
    15 that we know of, so far. People outside the HN crowd just started using AI pretty recently. There’s also a case to be made that the increased pollution has a certain death toll as well. We know that fossil fuels, construction, manufacturing, and even noise all produce pollution that reduce the average lifespan in a given area. Some of these data centers are being run off gas turbines which are very dirty.
JasonADrury 9mo ago
With half the planet using these, that hardly seems like many.

lostmsu 9mo ago

> Journalistic accounts describe individuals who have developed strong beliefs that chatbots are sentient But chatbots are sentient within a single context session!

seamossfet 9mo ago

They missed an opportunity to call this "cyber psychosis"

Lapsa 9mo ago

given secret service is capable of reading thoughts - channeling delusions ain't that much of a far stretch

ottah 9mo ago

This isn't psychosis, this is a cult. It's new age religious beliefs, wrapped up in tech. These people aren't any different than people who believe in speaking in tongues, or transubstantiation.