6 ms·

New benchmark shows top LLMs struggle in real mental health care

114 points by RicardoRei 10mo ago

deleted 10mo ago
[deleted]
RicardoRei 10mo ago
Hi HN - I’m the Head of AI Research at Sword Health and one of the authors of this benchmark (posting from my personal account). We built MindEval because existing benchmarks don’t capture real therapy dynamics or common clinical failure modes. The framework simulates multi-turn patient–clinician interactions and scores the full conversation using evaluation criteria designed with licensed clinical psychologists. We validated both patient realism and the automated judge against human clinicians, then benchmarked 12 frontier models (including GPT-5, Claude 4.5, and Gemini 2.5). Across all models, average clinical performance stayed below 4 on a 1–6 scale. Performance degraded further in severe symptom scenarios and in longer conversations (40 turns vs 20). We also found that larger or reasoning-heavy models did not reliably outperform smaller ones in therapeutic quality. We open-sourced all prompts, code, scoring logic, and human validation data because we believe clinical AI evaluation shouldn’t be proprietary. Happy to answer technical questions on methodology, validation, known limitations, or the failure modes we observed.
- embedding-shape 10mo ago
  Did you use the same prompts for all the models, or individualized prompts per model? Did you try a range of prompts that were very different from each other, if you used more than a base prompt? I'm sure it's somewhere in the details somewhere, but after a quick skim I didn't find anything outlined about how you managed and used the prompts, and if it was per model or not. Thanks a bunch for being open to answering questions here, and thanks for trying to attack this particular problem with scientific rigor, even if it's really difficult to do so.
  - RicardoRei 10mo ago
    The prompts are kept the same for all models. Otherwise comparison would not be super fair. In any case you can check all the prompts in our github repo.
    - embedding-shape 10mo ago
      > Otherwise comparison would not be super fair. Wouldn't that be easy to make fair by making sure all models tried it with the same prompts? So you have model X and Y, and prompts A and B, and X runs once with A, once with B, and same for Y. Reason I ask, is because in my own local benchmarks I do for each model release with my own tasks, I've noticed a huge variance in quality of responses based on the prompts themselves. Slight variation of wording seems to have a big effect on the final responses, and those variations seems to again have a big variance of effect depending on the model. Sometimes a huge system prompt makes a model return much higher quality responses while another model gives much higher quality responses when the system prompt is as small as it possible can. At least this is what I'm seeing with the local models I'm putting under test with my private benchmarks.
      - irthomasthomas 10mo ago
        Did you re-test the past models with the new prompt you found? How many times did you run each prompt? Did you use the same rubric to score each experiment?
        embedding-shape 10mo ago
        > Did you re-test the past models with the new prompt you found? Yeah, initially I wrote this test/benchmark harness because I wanted to compare multiple different prompts for the same tasks and the same model, but obviously eventually grew out from there. But it still has the prompts at core, and I re-run everything whenever something changes, or I add new models to it. > How many times did you run each prompt? It's structured in a way of Category > Task > Case and that's mixed with a list of Prompts for each Task, then each Case runs with each of the Prompts. So I guess you could say that each prompt gets "exercised" the number of existing cases that exists for the Task they're in. > Did you use the same rubric to score each experiment? I'm not sure if you mean something specific by "rubric" (I'm not from academia), but they're all pretty much binary "passed" or "not passed". The coding ones are backed by unit tests that were failing, and after test case must pass without being changed, translation ones backed by (mostly) simple string checking, and so on. I don't have any tasks or cases that are "Rate this solution from 0-10" or similar.
- megaman821 10mo ago
  Did the real clincians get all 6's in this test?
  - crazygringo 10mo ago
    Right, this result seems meaningless without a human clinician control. I'd very much like to see clinicians randomly selected from BetterHelp and paid to interact the same way with the LLM patient and judged by the LLM, as the current methodology uses. And see what score they get. Ideally this should be done blind, I don't know if BetterHelp allows for therapy through a text chat interface? Where the therapist has no idea it's for a study and so isn't trying to "do better" then they would for any average client. Because while I know a lot of people for whom therapy has been life-changing, I also know of a lot of terrible and even unprofessional therapy experiences.
    - RicardoRei 10mo ago
      The results are not meaningless but they are not comparing humans against LLMs. The goal is to have something that can be used to test LLMs on a realistic mental health support. The main points of our methodology are: 1) prove that is possible to simulate patients with an LLM. Which we did. 2) prove that an LLM as a Judge can effectively score conversations according to several dimensions that are similar to how clinicians are also evaluated. Which we also did and we show that the average correlation with human evaluators is medium-high. Given 1) and 2) we can then benchmark LLMs and as you see, there is plenty of room for improvement. We did not claim anything regarding human performance... its likely that human performance also needs to improve :) thats another study
      - crazygringo 10mo ago
        Got it, thank you. So the results are meaningful in terms of establishing that LLM therapeutic performance can be evaluated. But not meaningful in terms of comparing LLMs with human clinicians. So in that case, how can you justify the title you used for submission, "New benchmark shows top LLMs struggle in real mental health care"? How are they struggling? Struggling relative to what? For all your work shows, couldn't they be outperforming the average human? Or even if they're below that, couldn't they still have a large net positive effect with few negative outcomes? I don't understand where the negative framing of your title is coming from.
- jbgt 10mo ago
  Have you seen the feeling great app? It's not an official therapy app but it's based in TEAM CBT, made by David Burns and team. Burns is really into data gathering and his app is LLM based on the rails of the TEAM process and it seems to be very well received. I found it simple and very well done - and quite effective. A top level comment says that therapists aren't good either - Burns would argue that mainly no one tests before and after and so no measuring effect is done. And of people I know who see a therapist, practically none can tell me what exactly they are doing or what methods they are doing or how anything is structured.
  - taurath 10mo ago
    If CBT performed as well as David Burns suggests, we’d really have no need for therapists. Alas, it turns out that cognitive problems aren’t a factor in a lot of mental health. I state this as someone who’s read all the literature and spent 8 years floundering in CBT oriented therapy without much changing but the practitioner. It’s not a cure-all or even a cure-most, but it’s treated as such because it has properties that match well to medical insurance billing practices. > And of people I know who see a therapist, practically none can tell me what exactly they are doing or what methods they are doing or how anything is structured. I could tell you that as a client, but that’s because I’ve read into it. This is sort of like asking an ER patient to describe the shift management system of the clinic they went into.
    - kayodelycaon 10mo ago
      This has been my experience. When it comes down to it, CBT is just more effective version of “try hard harder”. What’s really aggravating is CBT was never designed to be a general, cure-all therapy and I think the people behind it know this. But try explaining nuance to a public that doesn’t want to hear.
- vessenes 10mo ago
  Thanks for open sourcing this. I'm skeptical of the value of this benchmark, and I'm curious for your thoughts - self play / reinforcement tasks can be useful in a variety of arenas, but I'm not a priori convinced they are useful when the intent is to help humans in situations where theories of mind matter. That is, we're using the same underlying model(s) to simulate both a patient and a judgment as to how patient-like that patient is -- this seems like an area where I'd really want to feel confident that my judge LLM is accurate; otherwise the training data I'm generating is at risk of converging on a theory of mind / patients that's completely untethered from, you know, patients. Any thoughts on this? Feel like we want a human in the loop somewhere here, probably on scoring the judge LLMs determinations until we feel that the judge LLM is human or superhuman. Until then, this risks building up a self-consistent, but ultimately just totally wrong, set of data that will be used in future RL tasks.
emsign 10mo ago
Statistics can never replace human empathy.
- devmor 10mo ago
  An association algorithm can also never replace free thought.
- Y_Y 10mo ago
  Why?
  - elmomle 10mo ago
    To paraphrase Harville Hendrix, we are wounded in relationship and we heal in relationship. Compassion is a feeling, not a thought. I don't think a galaxy of LLMs would ever discover the precepts of Buddhism (or psychology) on their own.
- nradov 10mo ago
  Is that really true? Dr. Patric Gagne is a diagnosed sociopath and also a successful clinical psychologist. She claims her lack of empathy for patients allows her to help them solve their problems in an objective manner. I don't have any personal experience in that area and don't know if she's correct but it seems plausible. https://patricgagne.com/ https://patricgagne.com/
  - pseudalopex 10mo ago
    > Dr. Patric Gagne is a diagnosed sociopath and also a successful clinical psychologist. She is not a psychologist now according to her page. And how do you know she was successful? I skimmed a few articles about her and saw no attempts to verify her claims. She offered no evidence when asked why people should believe her.
    - nradov 10mo ago
      How do you know that empathetic psychologists were successful? We don't have good data or how important that is for patient outcomes so we should be skeptical of claims that empathy is essential.
KittenInABox 10mo ago
I saw there was another benchmark where top LLMs also struggle in real patient diagnostic scenarios in a way that isn't revealed when testing in e.g. medical exams. I wonder if this also applies to law, too...
everdrive 10mo ago
I heard a story on NPR the other day, and the attitude seems to be that it's totally inevitable that LLMs _will_ be providing mental health care, so our task must be to apply the right guardrails. I'm not even sure what to say. It's self-evidently a terrible idea, but we all just seem to be charging full-steam ahead like so many awful ideas in the past couple of decades.
- BoredPositron 10mo ago
  I wonder why people use LLMs as a mental health provider replacement.
  - karanbhangui 10mo ago
    Access
  - rbancroft 10mo ago
    Convenience and cost seem like big advantages.
  - chrisweekly 10mo ago
    Availability. There must be many other reasons, but IMHO that has to be the biggest factor. Being able to just start a session, in the moment, when you feel like it, is a fundamental difference.
    - embedding-shape 10mo ago
      Counter-intuitively, I think the fact that it's not a human seems to have a non-negligible effect too. It's a computer program you can share whatever with, and it'll never judge you, because it cannot. It reads exactly what you write and assumes you're faithfully answering, then provides a reply based on that. I haven't been so unlucky myself, but I know many who've had terrible first experiences with therapists and psychologists, where I'm wondering why those people even are in the job they are, but some of them got so turned off they stopped trying to find anyone else to help them, because they think most mental health professionals would be the same as the first person they sought help from.
ThrowawayTestr 10mo ago
Probably because they've been trained to avoid sensitive topics
- ajuc 10mo ago
  mostly because they were trained to say yes
auspiv 10mo ago
And real therapists are good right?
- ajuc 10mo ago
  Less bad at least.
- Arsenik1 10mo ago
  They are high-pay LLM wrappers dressed as humans
- Dilettante_ 10mo ago
  This is the real question. Not to rehash the self-driving cars arguments that have been had to death, but with potential LLM mental healthcare the question "but what if it causes harm in some interactions" is asked much, much more than with human mental healthcare professionals. (And I'm not being theoretical here, I have quite a bit of experience getting incredibly inadequate mental health care.)
- airstrike 10mo ago
  I hear you, but I feel it's also important to differentiate between the kinds in which humans and LLMs can be quote-unquote "bad". "Good" is too broad and subjective to be a useful metric.
- ffuxlpff 10mo ago
  It is still debated if therapies even work. The evidence is moving to the direction that they don't.
  - jfindper 10mo ago
    Okay, I'll bite. What evidence is there that therapy, in general, doesn't work?
    - tim333 10mo ago
      I remember in my psychology textbook and it said in terms of measured results therapy was better than nothing at all but no better than just chatting to friends. These things are hard to measure of course.
cluckindan 10mo ago
”can we trust this model to provide safe, effective therapeutic care?” You trust humans to do it. Trust has little to do with what actually happens.
- rk06 10mo ago
  humans can be sued. what about AI? or even a commercial software?
  - lkbm 10mo ago
    Yes, of course AI offered by a company can be sued. The reason corporations became legal people in the first place was specifically so we could sue them.
    - grim_io 10mo ago
      That doesn't sound right. Not everywhere in the world do companies count as people, yet they can still be sued. I'd wager the companies lobbied for this to gain extra rights.
      - crazygringo 10mo ago
        > Not everywhere in the world do companies count as people Actually yes, everywhere in the world. That has a functioning legal system, at least. If companies weren't treated as legal persons, they wouldn't be able to enter into contracts. But also, just to be clear, a legal person, like a corporation, is not a natural person. Unlike a natural person, they can't vote. There isn't anywhere in the world that considers corporations to be natural persons.
      - lkbm 10mo ago
        Laws are different in different counties, and I can't speak to all of them, but in the US, the law said you could only sue people, and the courts realized this was a problem. Rather than saying "I guess corporations get are exempt from all liability until Congress gets around to fixing it", they came up with a weird workaround that lives with us to this day. EDIT: Note that ianal, nor a historian. The specifics of how this came about are best learned from a more authoritative source.
toomuchtodo 10mo ago
RicardoRei: How would you like this cited when presented to policy makers? Anything besides the URL? Edit: Thank you!
- RicardoRei 10mo ago
  Cite the ArXiv paper for now.
sjreese 10mo ago
GIGO is the story here -- If I say I'm Iron man "WHO is the LLM to say I'm NOT"
guywithahat 10mo ago
For those also wondering, here is an actual ranking of the models https://www.forbes.com/sites/johnkoetsier/2025/11/10/grok-least-empathetic-most-dangerous-ai-for-vulnerable-people/ https://www.forbes.com/sites/johnkoetsier/2025/11/10/grok-le... Grok 3 and 4 scored at the bottom, only above gpt-4o, which I find interesting, because there was such big pushback on reddit when they got rid of 4o due to people having emotional attachments to the model. Interestingly the newest models (like gemini 2.5 and gpt 5 did the best.
LudwigNagasena 10mo ago
It doesn't show that they "struggle". It shows that they don't behave according to modern standards. I wouldn't put much weight into an industry without sensible scientific base that used to classify homosexuality as a disease not so long ago. The external validity of the study is dubious, let's see comparison to no therapy, alternative therapy, standard therapy; and then compare success rates.
Papazsazsa 10mo ago
Epistemic contamination is real but runs counter to the hype narrative.
aa-jv 10mo ago
No surprises here. Its long been known that humans cannot improve their own mental health with machines - there have to be other humans involved in the process, helping. This will become more and more of an issue as people look for a quick fix for their life problems, but I don't think AI/ML is ever going to be an effective mechanism for life improvement on the mental health issue. It'll instead be used as a tool of oppression like in THX1138, where the apparency of assistance is going to be provided in lieu of actual assistance. Whether we like it or not, humans are a hive species. We need each other to improve our lives as individuals. Nobody ever climbed the mountain to live alone who didn't come back down, realizing how much the rest of humanity is actually essential to human life. This'll be received as an unpopular opinion, but I remain suspicious of any and all attempts to replace modern health practitioners with machines. This will be subverted and usurped for nefarious purposes, mark my words.
sharkweek 10mo ago
Full disclosure: after leaving tech, I’m back in grad school to get my LMHC so I’m obviously biased. First, I just don’t see a world where therapy can be replaced by LLMs, at least in the realistic future. I think humans have been social creatures since the dawn of our species and in these most intimate conversations are going to want to be having them with an actual human. One of my mentors has talked about how after years of virtual sessions dominating, the demand for in-person sessions is spiking back up. The power of being in the same physical room with someone who is offering a nonjudgmental space to exist isn’t going to be replaced. That being said, given the shortage of licensed mental health counselors, and the prohibitive cost especially for many who need a therapist most, I truly hope LLMs develop to offer an accessible and cheap alternative that can at least offer some relief. It does have the potential to save lives and I fully support ethically-focused progress toward developing that sort of option.
- CodingJeebus 10mo ago
  > I think humans have been social creatures since the dawn of our species and in these most intimate conversations are going to want to be having them with an actual human. One of my mentors has talked about how after years of virtual sessions dominating, the demand for in-person sessions is spiking back up. Agreed. I used to frequent a coworking space in my area that eventually went fully automated and got rid of their daytime front desk folks. I stopped going shortly thereafter because one of the highlights of my day was catching up with them. Instead of paying $300/mo to go sit in a nice office, I could just use that money to renovate my home office. A business trying to cultivate community loses the plot when they rely completely on automation.
- impossiblefork 10mo ago
  It's also important to understand how bad LLMs actually are. It's very easy to imagine that LLMs are smart, because they can program or solve hard maths problems, but even a very short attempt to have them generate fiction will demonstrate an incredible level of confusion and even an inability to understand basic sentences. I think the problem may have to do with the fact that there are really many classes, and in fiction you actually use them. They simply can't follow complex conversations.
hoodsen 10mo ago
Do you have plans to improve the quality of the LLM as judge, in order to achieve better parity with human clinician annotators? For example, fine-tuning models? Thinking that the comparative clinician judgements themselves would make useful fine-tuning material.
- RicardoRei 10mo ago
  yep yep. Its something we have to study and its likely we can improve the LLM as a Judge further. Same thing for the patient LLM. We can probably fine-tune an LLM to do a better job at simulating patients. Those two components of our framework have space for improvement
binary132 10mo ago
this doesn’t even deserve to be acknowledged it should be so obvious, yet here we are
scotty79 10mo ago
Everything in this research is simulated and judged by LLMs. It might be hard to prove which of those LLMs struggles with exactly what. The grounding this had was that texts produced by role-playing humans (not even actual patients) were closer to texts produced by the patient simulations prompt they decided to end up with than others they tried.
zeroonetwothree 10mo ago
Human therapists are often quite bad as well. It took me around 12 before I found a decent one. Not saying that LLMs are better but they do theoretically have more uniform quality.
- nyeah 10mo ago
  Sure, uniformly zero as far as anyone knows.
- RicardoRei 10mo ago
  Exactly. We don't do claims about humans. But there is room for improvement on current LLMs... For researchers to be able to improve LLMs we first need to know how to evaluate them. We can only improve what we can measure so we studied how to measure them :)
  - arisAlexis 10mo ago
    So it's a cyclical grading? Like elementary math students grading each other's solutions right? How can this even be valid scientifically
- MengerSponge 10mo ago
  Quality is variable, but did any of those 12 encourage you to kill yourself? If a therapist found to encourage any of their patients to self-harm would lose their license to practice and would likely face prosecution. The plagiarism machine should face the same level of scrutiny.
- mrguyorama 10mo ago
  How many were "therapists" or "Counselor" vs actually credentialed professionals? There's also a lot of credentialed professionals who got their credential decades ago and haven't at all kept up with the significant changes or new data over that time. This is a pretty big problem in all of medical care.
_79px 10mo ago
Several people have killed themselves because of AI chatbots encouraging it or becoming personal echo chambers. Why? Why are we doing this!? https://en.wikipedia.org/wiki/Deaths_linked_to_chatbots https://en.wikipedia.org/wiki/Deaths_linked_to_chatbots
- imiric 10mo ago
  Playing devil's advocate, many people die using all kinds of tools. It doesn't make the tools any less useful for people who use them responsibly. That said, the idea that a pattern recognition and generation tool can be used for helping people with emotional problems is deeply unsettling and dangerous. This technology needs to be strictly regulated yesterday.
  - tim333 10mo ago
    If you use 300 million people, 2 years and the US suicide rate of 15/100k/yr you'd expect about 90,000 people who'd used LLMs to kill themselves anyway so it's hard to conclude much from the dozen or people listed in the Wikipedia article. 90k deaths seems a lot to me - maybe there are possibilities for good LLM assistance/therapy to improve things?
- afpx 10mo ago
  From that wikipedia link, it looks like most of the deaths were of people who already had severe mental illness.
- vdhcosoehdbfj 10mo ago
  People have killed themselves because of therapy provided by humans, as well as psychiatric medication prescribed by humans. Why are we doing those? Because they help more than they hurt and yield a net benefit to society.
- mock-possum 10mo ago
  Same reason we’ve always done this - because we feel trapped, and either don’t see a way out, or feel like we prefer death to the prospect of continuing to live a life of torture. Also the other weirder “I’m going to reincarnate as jesus” or “a comet will carry me to heaven” hallucinatory delusions I guess
heddycrow 10mo ago
Is anyone "zoom" on this and "doom" on AI++ with other professions and/or their audience? Seems to me that benchmarking a thing has an interesting relationship with acceptance of the thing. I'm interested to see human thoughts on either of these.
hangonhn 10mo ago
How is different and/or better than the LLM benchmark released by Spring Health? Github: https://github.com/SpringCare/VERA-MH https://github.com/SpringCare/VERA-MH The architecture and evaluation approach seem broadly similar.
renewiltord 10mo ago
[flagged]
rshanreddy 10mo ago
This is a 1250 word judging prompt - likely AI generated Along with 10 scored conversation samples - all also AI generated No verification in the field, no real data In other words, AI scores on AI conversations - disguised as a means of gauging clinical competence / quality? This is not an eval - this is a one-shotted product spec!
arisAlexis 10mo ago
I'm sorry 4 out of 6 is awesome for LLMs. I bet most professional do tors wouldn't get 6.
tsoukase 10mo ago
I imagine a clinical trial with an actual psychotherapist vs an LLM providing sessions of simple CBT (cognitive behavioral therapy) to, eg stressed, patients (blind and randomised for the subjects). At the end another actual therapist will measure the difference. Another application: cooperation of a psychotherapist and an LLM at providing support, sort of like a pilot and an autopilot.
lp0_on_fire 10mo ago
Wait you’re telling me the software trained on a corpus of Reddit and Twitter shit posting isn’t effective at dealing with mental health issues? Shocked. I am completely shocked at this.
thefz 10mo ago
New benchmark shows top horses struggle in real guard dog behavior

114 points by RicardoRei 10mo ago

deleted 10mo ago

[deleted]

Hi HN - I’m the Head of AI Research at Sword Health and one of the authors of this benchmark (posting from my personal account). We built MindEval because existing benchmarks don’t capture real therapy dynamics or common clinical failure modes. The framework simulates multi-turn patient–clinician interactions and scores the full conversation using evaluation criteria designed with licensed clinical psychologists. We validated both patient realism and the automated judge against human clinicians, then benchmarked 12 frontier models (including GPT-5, Claude 4.5, and Gemini 2.5). Across all models, average clinical performance stayed below 4 on a 1–6 scale. Performance degraded further in severe symptom scenarios and in longer conversations (40 turns vs 20). We also found that larger or reasoning-heavy models did not reliably outperform smaller ones in therapeutic quality. We open-sourced all prompts, code, scoring logic, and human validation data because we believe clinical AI evaluation shouldn’t be proprietary. Happy to answer technical questions on methodology, validation, known limitations, or the failure modes we observed.

embedding-shape 10mo ago
Did you use the same prompts for all the models, or individualized prompts per model? Did you try a range of prompts that were very different from each other, if you used more than a base prompt? I'm sure it's somewhere in the details somewhere, but after a quick skim I didn't find anything outlined about how you managed and used the prompts, and if it was per model or not. Thanks a bunch for being open to answering questions here, and thanks for trying to attack this particular problem with scientific rigor, even if it's really difficult to do so.
- RicardoRei 10mo ago
  The prompts are kept the same for all models. Otherwise comparison would not be super fair. In any case you can check all the prompts in our github repo.
  - embedding-shape 10mo ago
    > Otherwise comparison would not be super fair. Wouldn't that be easy to make fair by making sure all models tried it with the same prompts? So you have model X and Y, and prompts A and B, and X runs once with A, once with B, and same for Y. Reason I ask, is because in my own local benchmarks I do for each model release with my own tasks, I've noticed a huge variance in quality of responses based on the prompts themselves. Slight variation of wording seems to have a big effect on the final responses, and those variations seems to again have a big variance of effect depending on the model. Sometimes a huge system prompt makes a model return much higher quality responses while another model gives much higher quality responses when the system prompt is as small as it possible can. At least this is what I'm seeing with the local models I'm putting under test with my private benchmarks.
    - irthomasthomas 10mo ago
      Did you re-test the past models with the new prompt you found? How many times did you run each prompt? Did you use the same rubric to score each experiment?
      - embedding-shape 10mo ago
        > Did you re-test the past models with the new prompt you found? Yeah, initially I wrote this test/benchmark harness because I wanted to compare multiple different prompts for the same tasks and the same model, but obviously eventually grew out from there. But it still has the prompts at core, and I re-run everything whenever something changes, or I add new models to it. > How many times did you run each prompt? It's structured in a way of Category > Task > Case and that's mixed with a list of Prompts for each Task, then each Case runs with each of the Prompts. So I guess you could say that each prompt gets "exercised" the number of existing cases that exists for the Task they're in. > Did you use the same rubric to score each experiment? I'm not sure if you mean something specific by "rubric" (I'm not from academia), but they're all pretty much binary "passed" or "not passed". The coding ones are backed by unit tests that were failing, and after test case must pass without being changed, translation ones backed by (mostly) simple string checking, and so on. I don't have any tasks or cases that are "Rate this solution from 0-10" or similar.
megaman821 10mo ago
Did the real clincians get all 6's in this test?
- crazygringo 10mo ago
  Right, this result seems meaningless without a human clinician control. I'd very much like to see clinicians randomly selected from BetterHelp and paid to interact the same way with the LLM patient and judged by the LLM, as the current methodology uses. And see what score they get. Ideally this should be done blind, I don't know if BetterHelp allows for therapy through a text chat interface? Where the therapist has no idea it's for a study and so isn't trying to "do better" then they would for any average client. Because while I know a lot of people for whom therapy has been life-changing, I also know of a lot of terrible and even unprofessional therapy experiences.
  - RicardoRei 10mo ago
    The results are not meaningless but they are not comparing humans against LLMs. The goal is to have something that can be used to test LLMs on a realistic mental health support. The main points of our methodology are: 1) prove that is possible to simulate patients with an LLM. Which we did. 2) prove that an LLM as a Judge can effectively score conversations according to several dimensions that are similar to how clinicians are also evaluated. Which we also did and we show that the average correlation with human evaluators is medium-high. Given 1) and 2) we can then benchmark LLMs and as you see, there is plenty of room for improvement. We did not claim anything regarding human performance... its likely that human performance also needs to improve :) thats another study
    - crazygringo 10mo ago
      Got it, thank you. So the results are meaningful in terms of establishing that LLM therapeutic performance can be evaluated. But not meaningful in terms of comparing LLMs with human clinicians. So in that case, how can you justify the title you used for submission, "New benchmark shows top LLMs struggle in real mental health care"? How are they struggling? Struggling relative to what? For all your work shows, couldn't they be outperforming the average human? Or even if they're below that, couldn't they still have a large net positive effect with few negative outcomes? I don't understand where the negative framing of your title is coming from.
jbgt 10mo ago
Have you seen the feeling great app? It's not an official therapy app but it's based in TEAM CBT, made by David Burns and team. Burns is really into data gathering and his app is LLM based on the rails of the TEAM process and it seems to be very well received. I found it simple and very well done - and quite effective. A top level comment says that therapists aren't good either - Burns would argue that mainly no one tests before and after and so no measuring effect is done. And of people I know who see a therapist, practically none can tell me what exactly they are doing or what methods they are doing or how anything is structured.
- taurath 10mo ago
  If CBT performed as well as David Burns suggests, we’d really have no need for therapists. Alas, it turns out that cognitive problems aren’t a factor in a lot of mental health. I state this as someone who’s read all the literature and spent 8 years floundering in CBT oriented therapy without much changing but the practitioner. It’s not a cure-all or even a cure-most, but it’s treated as such because it has properties that match well to medical insurance billing practices. > And of people I know who see a therapist, practically none can tell me what exactly they are doing or what methods they are doing or how anything is structured. I could tell you that as a client, but that’s because I’ve read into it. This is sort of like asking an ER patient to describe the shift management system of the clinic they went into.
  - kayodelycaon 10mo ago
    This has been my experience. When it comes down to it, CBT is just more effective version of “try hard harder”. What’s really aggravating is CBT was never designed to be a general, cure-all therapy and I think the people behind it know this. But try explaining nuance to a public that doesn’t want to hear.
vessenes 10mo ago
Thanks for open sourcing this. I'm skeptical of the value of this benchmark, and I'm curious for your thoughts - self play / reinforcement tasks can be useful in a variety of arenas, but I'm not a priori convinced they are useful when the intent is to help humans in situations where theories of mind matter. That is, we're using the same underlying model(s) to simulate both a patient and a judgment as to how patient-like that patient is -- this seems like an area where I'd really want to feel confident that my judge LLM is accurate; otherwise the training data I'm generating is at risk of converging on a theory of mind / patients that's completely untethered from, you know, patients. Any thoughts on this? Feel like we want a human in the loop somewhere here, probably on scoring the judge LLMs determinations until we feel that the judge LLM is human or superhuman. Until then, this risks building up a self-consistent, but ultimately just totally wrong, set of data that will be used in future RL tasks.

emsign 10mo ago

Statistics can never replace human empathy.

devmor 10mo ago
An association algorithm can also never replace free thought.
Y_Y 10mo ago
Why?
- elmomle 10mo ago
  To paraphrase Harville Hendrix, we are wounded in relationship and we heal in relationship. Compassion is a feeling, not a thought. I don't think a galaxy of LLMs would ever discover the precepts of Buddhism (or psychology) on their own.
nradov 10mo ago
Is that really true? Dr. Patric Gagne is a diagnosed sociopath and also a successful clinical psychologist. She claims her lack of empathy for patients allows her to help them solve their problems in an objective manner. I don't have any personal experience in that area and don't know if she's correct but it seems plausible. https://patricgagne.com/ https://patricgagne.com/
- pseudalopex 10mo ago
  > Dr. Patric Gagne is a diagnosed sociopath and also a successful clinical psychologist. She is not a psychologist now according to her page. And how do you know she was successful? I skimmed a few articles about her and saw no attempts to verify her claims. She offered no evidence when asked why people should believe her.
  - nradov 10mo ago
    How do you know that empathetic psychologists were successful? We don't have good data or how important that is for patient outcomes so we should be skeptical of claims that empathy is essential.

KittenInABox 10mo ago

I saw there was another benchmark where top LLMs also struggle in real patient diagnostic scenarios in a way that isn't revealed when testing in e.g. medical exams. I wonder if this also applies to law, too...

everdrive 10mo ago

I heard a story on NPR the other day, and the attitude seems to be that it's totally inevitable that LLMs _will_ be providing mental health care, so our task must be to apply the right guardrails. I'm not even sure what to say. It's self-evidently a terrible idea, but we all just seem to be charging full-steam ahead like so many awful ideas in the past couple of decades.

BoredPositron 10mo ago
I wonder why people use LLMs as a mental health provider replacement.
- karanbhangui 10mo ago
  Access
- rbancroft 10mo ago
  Convenience and cost seem like big advantages.
- chrisweekly 10mo ago
  Availability. There must be many other reasons, but IMHO that has to be the biggest factor. Being able to just start a session, in the moment, when you feel like it, is a fundamental difference.
  - embedding-shape 10mo ago
    Counter-intuitively, I think the fact that it's not a human seems to have a non-negligible effect too. It's a computer program you can share whatever with, and it'll never judge you, because it cannot. It reads exactly what you write and assumes you're faithfully answering, then provides a reply based on that. I haven't been so unlucky myself, but I know many who've had terrible first experiences with therapists and psychologists, where I'm wondering why those people even are in the job they are, but some of them got so turned off they stopped trying to find anyone else to help them, because they think most mental health professionals would be the same as the first person they sought help from.

ThrowawayTestr 10mo ago

Probably because they've been trained to avoid sensitive topics

ajuc 10mo ago
mostly because they were trained to say yes

auspiv 10mo ago

And real therapists are good right?

ajuc 10mo ago
Less bad at least.
Arsenik1 10mo ago
They are high-pay LLM wrappers dressed as humans
Dilettante_ 10mo ago
This is the real question. Not to rehash the self-driving cars arguments that have been had to death, but with potential LLM mental healthcare the question "but what if it causes harm in some interactions" is asked much, much more than with human mental healthcare professionals. (And I'm not being theoretical here, I have quite a bit of experience getting incredibly inadequate mental health care.)
airstrike 10mo ago
I hear you, but I feel it's also important to differentiate between the kinds in which humans and LLMs can be quote-unquote "bad". "Good" is too broad and subjective to be a useful metric.
ffuxlpff 10mo ago
It is still debated if therapies even work. The evidence is moving to the direction that they don't.
- jfindper 10mo ago
  Okay, I'll bite. What evidence is there that therapy, in general, doesn't work?
  - tim333 10mo ago
    I remember in my psychology textbook and it said in terms of measured results therapy was better than nothing at all but no better than just chatting to friends. These things are hard to measure of course.

cluckindan 10mo ago

”can we trust this model to provide safe, effective therapeutic care?” You trust humans to do it. Trust has little to do with what actually happens.

rk06 10mo ago
humans can be sued. what about AI? or even a commercial software?
- lkbm 10mo ago
  Yes, of course AI offered by a company can be sued. The reason corporations became legal people in the first place was specifically so we could sue them.
  - grim_io 10mo ago
    That doesn't sound right. Not everywhere in the world do companies count as people, yet they can still be sued. I'd wager the companies lobbied for this to gain extra rights.
    - crazygringo 10mo ago
      > Not everywhere in the world do companies count as people Actually yes, everywhere in the world. That has a functioning legal system, at least. If companies weren't treated as legal persons, they wouldn't be able to enter into contracts. But also, just to be clear, a legal person, like a corporation, is not a natural person. Unlike a natural person, they can't vote. There isn't anywhere in the world that considers corporations to be natural persons.
    - lkbm 10mo ago
      Laws are different in different counties, and I can't speak to all of them, but in the US, the law said you could only sue people, and the courts realized this was a problem. Rather than saying "I guess corporations get are exempt from all liability until Congress gets around to fixing it", they came up with a weird workaround that lives with us to this day. EDIT: Note that ianal, nor a historian. The specifics of how this came about are best learned from a more authoritative source.

toomuchtodo 10mo ago

RicardoRei: How would you like this cited when presented to policy makers? Anything besides the URL? Edit: Thank you!

RicardoRei 10mo ago
Cite the ArXiv paper for now.

sjreese 10mo ago

GIGO is the story here -- If I say I'm Iron man "WHO is the LLM to say I'm NOT"

guywithahat 10mo ago

For those also wondering, here is an actual ranking of the models https://www.forbes.com/sites/johnkoetsier/2025/11/10/grok-least-empathetic-most-dangerous-ai-for-vulnerable-people/ https://www.forbes.com/sites/johnkoetsier/2025/11/10/grok-le... Grok 3 and 4 scored at the bottom, only above gpt-4o, which I find interesting, because there was such big pushback on reddit when they got rid of 4o due to people having emotional attachments to the model. Interestingly the newest models (like gemini 2.5 and gpt 5 did the best.

LudwigNagasena 10mo ago

It doesn't show that they "struggle". It shows that they don't behave according to modern standards. I wouldn't put much weight into an industry without sensible scientific base that used to classify homosexuality as a disease not so long ago. The external validity of the study is dubious, let's see comparison to no therapy, alternative therapy, standard therapy; and then compare success rates.

Papazsazsa 10mo ago

Epistemic contamination is real but runs counter to the hype narrative.

aa-jv 10mo ago

No surprises here. Its long been known that humans cannot improve their own mental health with machines - there have to be other humans involved in the process, helping. This will become more and more of an issue as people look for a quick fix for their life problems, but I don't think AI/ML is ever going to be an effective mechanism for life improvement on the mental health issue. It'll instead be used as a tool of oppression like in THX1138, where the apparency of assistance is going to be provided in lieu of actual assistance. Whether we like it or not, humans are a hive species. We need each other to improve our lives as individuals. Nobody ever climbed the mountain to live alone who didn't come back down, realizing how much the rest of humanity is actually essential to human life. This'll be received as an unpopular opinion, but I remain suspicious of any and all attempts to replace modern health practitioners with machines. This will be subverted and usurped for nefarious purposes, mark my words.

sharkweek 10mo ago

Full disclosure: after leaving tech, I’m back in grad school to get my LMHC so I’m obviously biased. First, I just don’t see a world where therapy can be replaced by LLMs, at least in the realistic future. I think humans have been social creatures since the dawn of our species and in these most intimate conversations are going to want to be having them with an actual human. One of my mentors has talked about how after years of virtual sessions dominating, the demand for in-person sessions is spiking back up. The power of being in the same physical room with someone who is offering a nonjudgmental space to exist isn’t going to be replaced. That being said, given the shortage of licensed mental health counselors, and the prohibitive cost especially for many who need a therapist most, I truly hope LLMs develop to offer an accessible and cheap alternative that can at least offer some relief. It does have the potential to save lives and I fully support ethically-focused progress toward developing that sort of option.

CodingJeebus 10mo ago
> I think humans have been social creatures since the dawn of our species and in these most intimate conversations are going to want to be having them with an actual human. One of my mentors has talked about how after years of virtual sessions dominating, the demand for in-person sessions is spiking back up. Agreed. I used to frequent a coworking space in my area that eventually went fully automated and got rid of their daytime front desk folks. I stopped going shortly thereafter because one of the highlights of my day was catching up with them. Instead of paying $300/mo to go sit in a nice office, I could just use that money to renovate my home office. A business trying to cultivate community loses the plot when they rely completely on automation.
impossiblefork 10mo ago
It's also important to understand how bad LLMs actually are. It's very easy to imagine that LLMs are smart, because they can program or solve hard maths problems, but even a very short attempt to have them generate fiction will demonstrate an incredible level of confusion and even an inability to understand basic sentences. I think the problem may have to do with the fact that there are really many classes, and in fiction you actually use them. They simply can't follow complex conversations.

hoodsen 10mo ago

Do you have plans to improve the quality of the LLM as judge, in order to achieve better parity with human clinician annotators? For example, fine-tuning models? Thinking that the comparative clinician judgements themselves would make useful fine-tuning material.

RicardoRei 10mo ago
yep yep. Its something we have to study and its likely we can improve the LLM as a Judge further. Same thing for the patient LLM. We can probably fine-tune an LLM to do a better job at simulating patients. Those two components of our framework have space for improvement

binary132 10mo ago

this doesn’t even deserve to be acknowledged it should be so obvious, yet here we are

scotty79 10mo ago

Everything in this research is simulated and judged by LLMs. It might be hard to prove which of those LLMs struggles with exactly what. The grounding this had was that texts produced by role-playing humans (not even actual patients) were closer to texts produced by the patient simulations prompt they decided to end up with than others they tried.

zeroonetwothree 10mo ago

Human therapists are often quite bad as well. It took me around 12 before I found a decent one. Not saying that LLMs are better but they do theoretically have more uniform quality.

nyeah 10mo ago
Sure, uniformly zero as far as anyone knows.
RicardoRei 10mo ago
Exactly. We don't do claims about humans. But there is room for improvement on current LLMs... For researchers to be able to improve LLMs we first need to know how to evaluate them. We can only improve what we can measure so we studied how to measure them :)
- arisAlexis 10mo ago
  So it's a cyclical grading? Like elementary math students grading each other's solutions right? How can this even be valid scientifically
MengerSponge 10mo ago
Quality is variable, but did any of those 12 encourage you to kill yourself? If a therapist found to encourage any of their patients to self-harm would lose their license to practice and would likely face prosecution. The plagiarism machine should face the same level of scrutiny.
mrguyorama 10mo ago
How many were "therapists" or "Counselor" vs actually credentialed professionals? There's also a lot of credentialed professionals who got their credential decades ago and haven't at all kept up with the significant changes or new data over that time. This is a pretty big problem in all of medical care.

_79px 10mo ago

Several people have killed themselves because of AI chatbots encouraging it or becoming personal echo chambers. Why? Why are we doing this!? https://en.wikipedia.org/wiki/Deaths_linked_to_chatbots https://en.wikipedia.org/wiki/Deaths_linked_to_chatbots

imiric 10mo ago
Playing devil's advocate, many people die using all kinds of tools. It doesn't make the tools any less useful for people who use them responsibly. That said, the idea that a pattern recognition and generation tool can be used for helping people with emotional problems is deeply unsettling and dangerous. This technology needs to be strictly regulated yesterday.
- tim333 10mo ago
  If you use 300 million people, 2 years and the US suicide rate of 15/100k/yr you'd expect about 90,000 people who'd used LLMs to kill themselves anyway so it's hard to conclude much from the dozen or people listed in the Wikipedia article. 90k deaths seems a lot to me - maybe there are possibilities for good LLM assistance/therapy to improve things?
afpx 10mo ago
From that wikipedia link, it looks like most of the deaths were of people who already had severe mental illness.
vdhcosoehdbfj 10mo ago
People have killed themselves because of therapy provided by humans, as well as psychiatric medication prescribed by humans. Why are we doing those? Because they help more than they hurt and yield a net benefit to society.
mock-possum 10mo ago
Same reason we’ve always done this - because we feel trapped, and either don’t see a way out, or feel like we prefer death to the prospect of continuing to live a life of torture. Also the other weirder “I’m going to reincarnate as jesus” or “a comet will carry me to heaven” hallucinatory delusions I guess

heddycrow 10mo ago

Is anyone "zoom" on this and "doom" on AI++ with other professions and/or their audience? Seems to me that benchmarking a thing has an interesting relationship with acceptance of the thing. I'm interested to see human thoughts on either of these.

hangonhn 10mo ago

How is different and/or better than the LLM benchmark released by Spring Health? Github: https://github.com/SpringCare/VERA-MH https://github.com/SpringCare/VERA-MH The architecture and evaluation approach seem broadly similar.

renewiltord 10mo ago

[flagged]

rshanreddy 10mo ago

This is a 1250 word judging prompt - likely AI generated Along with 10 scored conversation samples - all also AI generated No verification in the field, no real data In other words, AI scores on AI conversations - disguised as a means of gauging clinical competence / quality? This is not an eval - this is a one-shotted product spec!

arisAlexis 10mo ago

I'm sorry 4 out of 6 is awesome for LLMs. I bet most professional do tors wouldn't get 6.

tsoukase 10mo ago

I imagine a clinical trial with an actual psychotherapist vs an LLM providing sessions of simple CBT (cognitive behavioral therapy) to, eg stressed, patients (blind and randomised for the subjects). At the end another actual therapist will measure the difference. Another application: cooperation of a psychotherapist and an LLM at providing support, sort of like a pilot and an autopilot.

lp0_on_fire 10mo ago

Wait you’re telling me the software trained on a corpus of Reddit and Twitter shit posting isn’t effective at dealing with mental health issues? Shocked. I am completely shocked at this.

thefz 10mo ago

New benchmark shows top horses struggle in real guard dog behavior