6 ms·

What a terrible, awful tragedy! A few months ago, OpenAI shared some data about how with 700 million users, 1 million people per week show signs of mental dist

What a terrible, awful tragedy! A few months ago, OpenAI shared some data about how with 700 million users, 1 million people per week show signs of mental distress in their chats [1]. OpenAI is aware of the problem [2], not doing enough, and they shouldn't be hiding data. (There is also a great NYT Magazine piece about a person who fell into AI Psychosis [3].) The links in other comments to Less Wrong posts attempting to dissuade people from thinking that they have "awoken their instance of ChatGPT into consciousness", or that they've made some breakthrough in "AI Alignment" without doing any real math (etc.) suggest that ChatGPT and other LLMs have a problem of reinforcing patterns of grandiose and narcissistic thinking. The problem is multiplied by the fact that it is all too easy for us (as a species) to collectively engage in motivated social cognition. Bill Hicks had a line about how if you were high on drugs and thought you could fly, maybe try taking off from the ground rather than jumping out of a window. Unfortunately, people who are engaging in motivated social cognition (also called identity protective cognition) and are convinced that they are having a divine revelation are not the kind of people who want to be correct and who are therefore open to feedback. Because one could "simply" ask a different LLM to neutrally evaluate the conversation / conversational snippets. I've found Gemini to be useful for a second or even third opinion. But this means that one would be happy to be told that one is wrong. [1] https://www.bmj.com/content/391/bmj.r2290.full https://www.bmj.com/content/391/bmj.r2290.full [2] https://openai.com/index/strengthening-chatgpt-responses-in-sensitive-conversations/ https://openai.com/index/strengthening-chatgpt-responses-in-... [3] https://www.nytimes.com/2025/08/08/technology/ai-chatbots-delusions-chatgpt.html https://www.nytimes.com/2025/08/08/technology/ai-chatbots-de...

DocTomoe 9mo ago
> A few months ago, OpenAI shared some data about how with 700 million users, 1 million people per week show signs of mental distress in their chats Considering that the global prevalence of mental health issues in the population is one in seven[1], that would make OpenAI users about 100 times more 'sane' than the general population. Either ChatGPT miraculously selects for an unusually healthy user base - or "showing signs of mental distress in chat logs" is not the same thing as being mentally ill, let alone harmed by the tool. [1] https://www.who.int/news-room/fact-sheets/detail/mental-disorders https://www.who.int/news-room/fact-sheets/detail/mental-diso...
- zahlman 9mo ago
  Having a mental health issue is not at all the same thing as "showing signs of mental distress" in any particular "chat". Many forms of mental illness wouldn't show up in dialogue normally; when it would, it doesn't necessarily show up all the time. And then there's the matter of detecting it in the transcript.
- tehjoker 9mo ago
  I don't know the full details, but 700M users and 1 million per a week, means up to 52M per year though I imagine a lot of them show up multiple weeks.
  - DocTomoe 9mo ago
    You also don't take into account that the userbase itself is shifting. That being said: Those of us who grew up when the internet was still young remember alt.suicide.holiday, and when you could buy books explaining relatively painless methods on amazon. People are depressed. It's a result of the way we choose to live as a civilization. Some don't make the cut. We should start accepting that. In fact, forcing people to live on in a world that is unsuited for happiness might constitute cruel and unusual punishment.
    - GreenWatermelon 9mo ago
      Maybe, just maybe, we should fix the fucked up world we created instead? Shunning the modern culture of individualism would be a great first step, followed by promoting communal culture. Live exactly how we evolved to live for hundreds of thousands of years.
gaigalas 9mo ago
Why do you think a breakthrough in AI Alignment should require doing math? Many alignment problems are solved not by math formulas, but by insights into how to better prepare training data and validation steps.
- YossarianFrPrez 9mo ago
  Fair question. While I'm not an expert on AI Alignment, I'd be surprised if any AI alignment approach did not involve real math at some point, given that all machine learning algorithms are inherently mathematical-computational in nature. Like I would imagine one has to know things like how various reward functions work, what happens in the modern variants of attention mechanisms, how different back-propagation strategies affect the overall result etc. in order to come up with (and effectively leverage) reinforcement learning with human feedback. I did a little searching, here's a 2025 review I found by entering "AI Alignment" into Google Scholar, and it has at least one serious looking mathematical equation: https://dl.acm.org/doi/full/10.1145/3770749 https://dl.acm.org/doi/full/10.1145/3770749 (section 2.2). This being said, maybe you have examples of historical breakthroughs in AI Alignment that didn't involve doing / understanding the mathematical concepts I mentioned in the previous paragraph? In the context of the above article, I think it's possible that some people are talking to ChatGPT on a buzzword level end up thinking that alignment can be solved via "fractal recursion of human in the loop validation sessions" for example. It seems like a modern incarnation of people thinking they can trisect the angle: https://www.ufv.ca/media/faculty/gregschlitt/information/WhatToDoWhenTrisectorComes.pdf https://www.ufv.ca/media/faculty/gregschlitt/information/Wha...
  - DenisM 9mo ago
    > maybe you have examples of historical breakthroughs in AI Alignment that didn't involve doing / understanding the mathematical concepts I mentioned in the previous paragraph? Multi agentic systems appear to have strong potential. Will that work out? I don’t know. But I know the potential there.
  - gaigalas 9mo ago
    > maybe you have examples of historical breakthroughs in AI Alignment OpenAI confessions is a good example of largely non-mathematical insight: https://arxiv.org/abs/2512.08093 https://arxiv.org/abs/2512.08093 I don't know, I think it's good stuff. Would you agree? > I think it's possible that some people are talking to ChatGPT on a buzzword level I never said this is not happening. This definitely happens. What I said is very different. I'm saying that you don't need to be a mathematician to have good insights into novel ways of improving AI alignment. You definitely need good epistemic intuition though.
JohnMakin 9mo ago
It's probably an artifact of how I use it (I turn off any kind of history or "remembering" of past conversations), but when I started becoming really impressed by tools like claude/chatgpt/etc. was the first time I was chasing down some dumb idea I had for work, convinced I was right, and it finally gently told me I was wrong (in its own way). That is exactly what I want these things to do, but it seems like most users do not want to be told they are wrong, and the companies are not very incentivized to encourage these tools to behave that way. I have identified very few instances where something like chatGPT just randomly started praising me (outside of the whole "you're absolutely correct to push back on this" kind of thing). I guess leading questions probably have something to do with this.
- okayGravity 9mo ago
  It all has to do with specific filler words you use when prompting, especially chatGPT. If you use words that suggest a heavy (and I mean you really have to make the LLM know you're questioning), then it will question to an extent as you imply. If you look at the chats that they do have from this incident, he phrased his prompts as more convincing rather than questioning (i.e "Shes doing this because of this!") So chatGPT roleplays and goes along with the delusion. Most people will just talk to LLMs like they are a person, even though LLMs won't understand the difference in complex social language and reasoning. It's almost like robots aren't people!
- Avamander 9mo ago
  In one recent thread about StackOverflow dying, some people theorized that the success of LLMs and thus failing of SO could mostly be attributed to the amount of sycophancy of LLMs. I tend to agree more and more. People need to be told when their ideas are wrong, if they like it or not.
  - StableAlkyne 9mo ago
    There's also the communications aspect: SO was/is a great site for getting information if (and only if) you properly phrase your question. Oftentimes, if you had an X/Y problem, you would quickly get corrected. God help you if you had an X/Y Problem Problem. Or if English wasn't your first language. I suspect the popularity is also boosted by the last two; it will happily tell you the best way to do whatever cursed thing you're trying to do, while still not judging over English skills.
baranul 9mo ago
Companies want the money and continual engagement. People getting addicted to AI, as trusted advisor or friend, is money in their pockets. Just like having people addicted to gambling or alcohol, it's all big business. It's becoming even more apparent, that there is a line between using AI as a tool to accomplish a task versus excessively relying on it for psychological reasons.
mmooss 9mo ago
> Because one could "simply" ask a different LLM to neutrally evaluate the conversation / conversational snippets. The problem is using LLMs beyond a limited scope, which is free ideas but not reliable reasoning or, goodness forbid, decision-making. Maybe the model for LLMs is a very good, sociopathic sophist or liar. They know a lot of 'facts', true or false, and are can con you out of your car keys (or house or job). Sometimes you catch them at a lie and their dishonesty becomes transparent. They have good ideas, though their usefulness only enhances their con jobs. (They also tell everything you say with others.) Would you rely on them for something of any importance? Simply ask a human.

DocTomoe 9mo ago

> A few months ago, OpenAI shared some data about how with 700 million users, 1 million people per week show signs of mental distress in their chats Considering that the global prevalence of mental health issues in the population is one in seven[1], that would make OpenAI users about 100 times more 'sane' than the general population. Either ChatGPT miraculously selects for an unusually healthy user base - or "showing signs of mental distress in chat logs" is not the same thing as being mentally ill, let alone harmed by the tool. [1] https://www.who.int/news-room/fact-sheets/detail/mental-disorders https://www.who.int/news-room/fact-sheets/detail/mental-diso...

zahlman 9mo ago
Having a mental health issue is not at all the same thing as "showing signs of mental distress" in any particular "chat". Many forms of mental illness wouldn't show up in dialogue normally; when it would, it doesn't necessarily show up all the time. And then there's the matter of detecting it in the transcript.
tehjoker 9mo ago
I don't know the full details, but 700M users and 1 million per a week, means up to 52M per year though I imagine a lot of them show up multiple weeks.
- DocTomoe 9mo ago
  You also don't take into account that the userbase itself is shifting. That being said: Those of us who grew up when the internet was still young remember alt.suicide.holiday, and when you could buy books explaining relatively painless methods on amazon. People are depressed. It's a result of the way we choose to live as a civilization. Some don't make the cut. We should start accepting that. In fact, forcing people to live on in a world that is unsuited for happiness might constitute cruel and unusual punishment.
  - GreenWatermelon 9mo ago
    Maybe, just maybe, we should fix the fucked up world we created instead? Shunning the modern culture of individualism would be a great first step, followed by promoting communal culture. Live exactly how we evolved to live for hundreds of thousands of years.

gaigalas 9mo ago

Why do you think a breakthrough in AI Alignment should require doing math? Many alignment problems are solved not by math formulas, but by insights into how to better prepare training data and validation steps.

YossarianFrPrez 9mo ago
Fair question. While I'm not an expert on AI Alignment, I'd be surprised if any AI alignment approach did not involve real math at some point, given that all machine learning algorithms are inherently mathematical-computational in nature. Like I would imagine one has to know things like how various reward functions work, what happens in the modern variants of attention mechanisms, how different back-propagation strategies affect the overall result etc. in order to come up with (and effectively leverage) reinforcement learning with human feedback. I did a little searching, here's a 2025 review I found by entering "AI Alignment" into Google Scholar, and it has at least one serious looking mathematical equation: https://dl.acm.org/doi/full/10.1145/3770749 https://dl.acm.org/doi/full/10.1145/3770749 (section 2.2). This being said, maybe you have examples of historical breakthroughs in AI Alignment that didn't involve doing / understanding the mathematical concepts I mentioned in the previous paragraph? In the context of the above article, I think it's possible that some people are talking to ChatGPT on a buzzword level end up thinking that alignment can be solved via "fractal recursion of human in the loop validation sessions" for example. It seems like a modern incarnation of people thinking they can trisect the angle: https://www.ufv.ca/media/faculty/gregschlitt/information/WhatToDoWhenTrisectorComes.pdf https://www.ufv.ca/media/faculty/gregschlitt/information/Wha...
- DenisM 9mo ago
  > maybe you have examples of historical breakthroughs in AI Alignment that didn't involve doing / understanding the mathematical concepts I mentioned in the previous paragraph? Multi agentic systems appear to have strong potential. Will that work out? I don’t know. But I know the potential there.
- gaigalas 9mo ago
  > maybe you have examples of historical breakthroughs in AI Alignment OpenAI confessions is a good example of largely non-mathematical insight: https://arxiv.org/abs/2512.08093 https://arxiv.org/abs/2512.08093 I don't know, I think it's good stuff. Would you agree? > I think it's possible that some people are talking to ChatGPT on a buzzword level I never said this is not happening. This definitely happens. What I said is very different. I'm saying that you don't need to be a mathematician to have good insights into novel ways of improving AI alignment. You definitely need good epistemic intuition though.

JohnMakin 9mo ago

It's probably an artifact of how I use it (I turn off any kind of history or "remembering" of past conversations), but when I started becoming really impressed by tools like claude/chatgpt/etc. was the first time I was chasing down some dumb idea I had for work, convinced I was right, and it finally gently told me I was wrong (in its own way). That is exactly what I want these things to do, but it seems like most users do not want to be told they are wrong, and the companies are not very incentivized to encourage these tools to behave that way. I have identified very few instances where something like chatGPT just randomly started praising me (outside of the whole "you're absolutely correct to push back on this" kind of thing). I guess leading questions probably have something to do with this.

okayGravity 9mo ago
It all has to do with specific filler words you use when prompting, especially chatGPT. If you use words that suggest a heavy (and I mean you really have to make the LLM know you're questioning), then it will question to an extent as you imply. If you look at the chats that they do have from this incident, he phrased his prompts as more convincing rather than questioning (i.e "Shes doing this because of this!") So chatGPT roleplays and goes along with the delusion. Most people will just talk to LLMs like they are a person, even though LLMs won't understand the difference in complex social language and reasoning. It's almost like robots aren't people!
Avamander 9mo ago
In one recent thread about StackOverflow dying, some people theorized that the success of LLMs and thus failing of SO could mostly be attributed to the amount of sycophancy of LLMs. I tend to agree more and more. People need to be told when their ideas are wrong, if they like it or not.
- StableAlkyne 9mo ago
  There's also the communications aspect: SO was/is a great site for getting information if (and only if) you properly phrase your question. Oftentimes, if you had an X/Y problem, you would quickly get corrected. God help you if you had an X/Y Problem Problem. Or if English wasn't your first language. I suspect the popularity is also boosted by the last two; it will happily tell you the best way to do whatever cursed thing you're trying to do, while still not judging over English skills.

baranul 9mo ago

Companies want the money and continual engagement. People getting addicted to AI, as trusted advisor or friend, is money in their pockets. Just like having people addicted to gambling or alcohol, it's all big business. It's becoming even more apparent, that there is a line between using AI as a tool to accomplish a task versus excessively relying on it for psychological reasons.

mmooss 9mo ago

> Because one could "simply" ask a different LLM to neutrally evaluate the conversation / conversational snippets. The problem is using LLMs beyond a limited scope, which is free ideas but not reliable reasoning or, goodness forbid, decision-making. Maybe the model for LLMs is a very good, sociopathic sophist or liar. They know a lot of 'facts', true or false, and are can con you out of your car keys (or house or job). Sometimes you catch them at a lie and their dishonesty becomes transparent. They have good ideas, though their usefulness only enhances their con jobs. (They also tell everything you say with others.) Would you rely on them for something of any importance? Simply ask a human.