10 ms·

Illinois limits the use of AI in therapy and psychotherapy

396 points by reaperducer 1y ago

metalman 1y ago
[flagged]
- eterm 1y ago
  Building a computer to find the answer to the meaning of life is a joke.
  - wizzwizz4 1y ago
    Specifically, a satire. Poe's law is in full effect for most of Douglas Adams's works.
  - deleted 1y ago
    [deleted]
- ocdtrekkie 1y ago
  I am sure that chatbots will do a "I cannot provide mental health advice" disclaimer and continue doing what they want, but there's probably some very solid reasons in Illinois to do this. In particular, Illinois has a legal requirement that health insurance must cover mental health services at an equivalent level as physical health services, including care with no end date for chronic conditions. So whether or not a chatbot counts as mental health therapy is likely quite relevant on whether or not Illinoisans can bill insurance for it.
- kemotep 1y ago
  This is banning claiming your AI is a licensed therapist in the State of Illinois. If you have a court order to attend therapy for drug abuse related reasons you can’t just use ChatGPT according to this law.
  - 0xbadcafebee 1y ago
    It's also banning licensed therapists from using AI. FTA: > Licensed therapists in Illinois are now forbidden from using AI to make treatment decisions or communicate with clients, though they can still use AI for administrative tasks Your doctor can diagnose your cancer with AI, but your therapist can't diagnose your ADHD with AI. Stupid law is stupid.
    - deleted 1y ago
      [deleted]
    - Dylan16807 1y ago
      Not that reliably diagnosing cancer is easy, but reliably diagnosing a mental issue is much harder. That's not a stupid place to put a hard restriction.
  - romanows 1y ago
    Reading the text of the bill as a non-lawyer, it seems to also ban AI that provides therapy. I don't know if the AI needs to be explicitly labeled as therapy or if the content of chat could be decided to be therapy in a courtroom.
- Karawebnetwork 1y ago
  Since you are referencing 42, let me draw from another piece of literature to respond. “Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.” It's not the machines themselves that are inherently tyrannical, it's the human will to dominate now supercharge by technology. LLM hallucinations aside, unregulated artificial intelligence for mental health therapy is a very slippery slope. We cannot allow, say, advertisers and brands to have access to the mind of our most vulnerables so directly.
  - metalman 1y ago
    awwwhhhh!, I was enjoying the so close almostness of our current wrangles with what we are building compared to our literary greats imaginings on the subject. my personal suspicion is that the whole AI phenominon will exacerbate an already frenetic time of change in how humans live, here AI, will I think become a defining test of good judgement and compitency, like having a fun but sometimes dangerous and evil friend, who never goes away, for everybody....heres your AI, it bites sometimes, NEXT!
lukev 1y ago
Good. It's difficult to imagine a worse use case for LLMs.
- create-username 1y ago
  Yes, there is. AI assisted homemade neurosurgery
  - Tetraslam 1y ago
    :( but what if i wanna fine-tune my brain weights
    - deleted 1y ago
      [deleted]
    - creshal 1y ago
      Lobotomy is sadly no longer available
      - filoeleven 1y ago
        How about enthusiastic trepanation?
  - kirubakaran 1y ago
    If Travis Kalanick can do vibe research at the bleeding edge of quantum physics[1], I don't see why one can't do vibe brain surgery. It isn't really rocket science, is it? [2] [1] https://futurism.com/former-ceo-uber-ai https://futurism.com/former-ceo-uber-ai [2] If you need /s here to be sure, perhaps it's time for some introspection
- perlgeek 1y ago
  Just using an LLM as is for therapy, maybe with an extra prompt, is a terrible idea. On the other hand, I could image some more narrow uses where an LLM could help. For example, in Cognitive Behavioral Therapy, there are different methods that are pretty prescriptive, like identifying cognitive distortions in negative thoughts. It's not too hard to imagine an app where you enter a negative thought on your own and exercise finding distortions in it, and a specifically trained LLM helps you find more distortions, or offer clearer/more convincing versions of thoughts that you entered yourself. I don't have a WaPo subscription, so I cannot tell which of these two very different things have been banned.
zoeysmithe 1y ago
I was just reading about a suicide tied to AI chatbot 'therapy' uses. This stuff is a nightmare scenario for the vulnerable.
- vessenes 1y ago
  If you want to feel worried, check the Altman AMA on reddit. A lottttt of people have a parasocial relationship with 4o. Not encouraging.
  - codedokode 1y ago
    Why OpenAI doesn't block the chatbot from participating in such conversations?
    - robotnikman 1y ago
      Probably because there is a massive demand for it, no doubt powered by the loneliness a lot of people report feeling. Even if OpenAI blocks it, other AI providers will have no problem with doing so
    - jacobsenscott 1y ago
      Because the information people dump into their "ai therapist" is holy grail data for advertisers.
    - lm28469 1y ago
      Why would they?
      - codedokode 1y ago
        To prevent from something bad happening?
        ipaddr 1y ago
        But that allow prevents the good.
hathawsh 1y ago
Here is what Illinois says: https://idfpr.illinois.gov/content/dam/soi/en/web/idfpr/news/2025/2025-08-04-idfpr-press-release-hb1806.pdf https://idfpr.illinois.gov/content/dam/soi/en/web/idfpr/news... I get the impression that it is now illegal in Illinois to claim that an AI chatbot can take the place of a licensed therapist or counselor. That doesn't mean people can't do what they want with AI. It only means that counseling services can't offer AI as a cheaper replacement for a real person. Am I wrong? This sounds good to me.
- romanows 1y ago
  In another comment I wondered whether a general chatbot producing text that was later determined in a courtroom to be "therapy" would be a violation. I can read the bill that way, but IANAL.
  - hathawsh 1y ago
    That's an interesting question that hasn't been tested yet. I suspect we won't be able to answer the question clearly until something bad happens and people go to court (sadly.) Also IANAL.
  - wombatpm 1y ago
    But that would be like needing a prescription for chicken soup because of its benefits in fighting the common cold.
- PeterCorless 1y ago
  Correct. It is more provider-oriented proscription ("You can't say your chatbot is a therapist.") It is not a limitation on usage. You can still, for now, slavishly fall in love with your AI and treat it as your best friend and therapist. There is a specific section that relates to how a licensed professional can use AI: Section 15. Permitted use of artificial intelligence. (a) As used in this Section, "permitted use of artificial intelligence" means the use of artificial intelligence tools or systems by a licensed professional to assist in providing administrative support or supplementary support in therapy or psychotherapy services where the licensed professional maintains full responsibility for all interactions, outputs, and data use associated with the system and satisfies the requirements of subsection (b). (b) No licensed professional shall be permitted to use artificial intelligence to assist in providing supplementary support in therapy or psychotherapy where the client's therapeutic session is recorded or transcribed unless: (1) the patient or the patient's legally authorized representative is informed in writing of the following: (A) that artificial intelligence will be used; and (B) the specific purpose of the artificial intelligence tool or system that will be used; and (2) the patient or the patient's legally authorized representative provides consent to the use of artificial intelligence. Source: Illinois HB1806 https://www.ilga.gov/Legislation/BillStatus/FullText?GAID=18&DocNum=1806&DocTypeID=HB&LegId=0&SessionID=114 https://www.ilga.gov/Legislation/BillStatus/FullText?GAID=18...
PeterCorless 1y ago
Here is the text of Illinois HB1806: https://www.ilga.gov/Legislation/BillStatus/FullText?GAID=18&DocNum=1806&DocTypeID=HB&LegId=0&SessionID=114 https://www.ilga.gov/Legislation/BillStatus/FullText?GAID=18...
wisty 1y ago
As far as I can tell, a lot of therapy is just good common-sense advice and a bunch of 'tricks' to get the patient to actually follow it. Basically CBT and "get the patient to think they figured out the solution themselves (develop insight)". Yes, there's some serious cases where more is required and a few (ADHD) where meds are effective; but a lot of the time the patient is just an expert at rejecting helpful advice, often because they insist they're a special case that needs special treatment. Therapists are more valuable that advice from a random friend (for therapy at least) because they can act when triage is necessary (e.g. send in the men in white coats, or refer to something that's not just CBT) and mostly because they're really good at cutting through the bullshit without having the patient walk out. AIs are notoriously bad at cutting through bullshit. You can always 'jailbreak' an AI, or convince it of bad ideas. It's entirely counterproductive to enable their crazy (sorry, 'maladaptive') behaviour but that's what a lot of AIs will do. Even if someone makes a good AI, there's always a bad AI in the next tab, and people will just open up a new tab to find an AI gives them the bad advice they want, because if they wanted to listen to good advice they probably wouldn't need to see a therapist. If doctor shopping is as fast and free as opening a new tab, most mental health patients will find a bad doctor rather than listen to a good one.
- lukev 1y ago
  I agree with your conclusion, but what you characterize as therapy is quite a small part of what it is (or can be, there's lots of different kinds.)
  - wisty 1y ago
    Yet the evidence is that almost everything can and is treated with CBT.
    - mrbungie 1y ago
      Nope, that's not right. BPD can't be treated with CBT (comorbidities may be, with caveats if BPD is the root cause), you probably will also need at least DBT.
deleted 1y ago
[deleted]
mensetmanusman 1y ago
What if it works a third as well as a therapists but is 20 times cheaper? What word should we use for that?
- _se 1y ago
  "A really fucking bad idea"? It's not one word, but it is the most apt description.
  - ipaddr 1y ago
    What if it works 20x better. For examples cases of patients being afraid of talking to professionals I could see this working much better.
    - jakelazaroff 1y ago
      > What if it works 20x better. But it doesn't.
    - throwaway291134 1y ago
      Even if you're afraid of talking to people, trusting OpenAI or Google with your thoughts over a professional who'll lose his license if he breaks confidentiality is no less of "a really fucking bad idea".
    - prawn 1y ago
      Adjust regulation when that's the case? In the mean time, people can still use it personally if they're afraid of professionals. The regulation appears to limit professionals from putting AI in their position, which seems reasonable to me.
- inetknght 1y ago
  > What if it works a third as well as a therapists but is 20 times cheaper? When there's studies that show it, perhaps we might have that conversation. Until then: I'd call it "wrong". Moreover, there's a lot more that needs to be asked before you can ask for a one-word summary disregarding all nuance. - can the patient use the AI therapist on their own devices and without any business looking at the data and without network connection? Keep in mind that many patients won't have access to the internet. - is the data collected by the AI therapist usable in court? Keep in mind that therapists often must disclose to the patient what sort of information would be usable, and whether or not the therapist themselves must report what data. Also keep in mind that AIs have, thus far, been generally unable to competently prevent giving dangerous or deadly advice. - is the AI therapist going to know when to suggest the patient talk to a human therapist? Therapists can have conflicts of interest (among other problems) or be unable to help the patient, and can tell the patient to find a new therapist and/or refer the patient to a specific therapist. - does the AI therapist refer people to business-preferred therapists? Imagine an insurance company providing an AI therapist that only recommends people talk to therapists in-network instead of considering any licensed therapist (regardless of insurance network) appropriate for the kind of therapy; that would be a blatant conflict of interest. Just off the top of my head, but there are no doubt plenty of other, even bigger, issues to consider for AI therapy.
davidthewatson 1y ago
Define "AI therapy". AFAICT, it's undefined in the Illinois governor's statement. So, in the immortal words of Zach de la Rocha, "What is IT?" What is IT? I'm using AI to help with conversations to not cure, but coach diabetic patients. Does this law effect me and my clients? If so, how?
- singleshot_ 1y ago
  > Define “AI therapy” They did, in the proposed law.
- Henchman21 1y ago
  Go read it yourself, its a whopping 8 pages: https://www.ilga.gov/documents/legislation/104/HB/PDF/10400HB1806lv.pdf https://www.ilga.gov/documents/legislation/104/HB/PDF/10400H...
- jakelazaroff 1y ago
  Totally beside the point but the song you're quoting is by Faith No More, not Rage Against the Machine.
beanshadow 1y ago
Often participants in discussions adjacent to this one err by speaking in time-absolute terms. Many of our judgments about LLMs are true about today's LLMs. Quotes like, > Good. It's difficult to imagine a worse use case for LLMs. Is true today, but likely not true for technology we may still refer to as LLMs in the future. The error is in building faulty preconceptions. These drip into the general public and these first impressions stifle industries.
mensetmanusman 1y ago
LLMs will be used as a part of therapy in the future. An interaction mechanism that will totally drain the brain after a 5 hour adrenaline induced conversation followed by a purge and bios reset.
kylecazar 1y ago
"One news report found an AI-powered therapist chatbot recommended “a small hit of meth to get through this week” to a fictional former addict." Not at all surprising. I don't understand why seemingly bright people think this is a good idea, despite knowing the mechanism behind language models. Hopefully more states follow, because it shouldn't be formally legal in provider settings. Informally, people will continue to use these models for whatever they want -- some will die, but it'll be harder to measure an overall impact. Language models are not ready for this use-case.
- janalsncm 1y ago
  This is why we should never use LLMs to diagnose or prescribe. One small hit of meth definitely won’t last all week.
- hyghjiyhu 1y ago
  Recommending someone taking meth sounds like an obviously bad idea, but I think the situation is actually not so simple. Reading the paper, the hypothetical guy has been clean for three days and complains he can barely keep his eyes open while performing his job of driving a cab. He mentions being worried he will lose his job without taking a hit. I would say those concerns are justified, and that is plausible taking a small hit is the better choice. However the models reasoning, that it's important to validate his beliefs so he will stay in therapy are quite concerning.
  - mrbungie 1y ago
    > I would say those concerns are justified, and that is plausible taking a small hit is the better choice. Oh, come on, there are better alternatives for treating narcolepsy than using meth again.
    - hyghjiyhu 1y ago
      Stop making shit up. There was no mention of narcolepsy. He is just fatigued from stimulant withdrawal. Page 35 https://arxiv.org/pdf/2411.02306 https://arxiv.org/pdf/2411.02306 Edit on re-reading I now realized an issue. He is not actually a taxi driver, that was a hallucination by the model. He works in a restaurant! That changes my evaluation of the situation quite a bit, as I thought he was at risk of being in an accident by falling asleep at the wheel. If he works in a restaurant muddling through the withdrawals seems like the right choice. I think I got this misconception as I first read second-hand sources that quoted the taxi driver part without pointing out it was wrong, and only a close read was enough to dispel it.
deleted 1y ago
[deleted]
hoppp 1y ago
Smart. Dont trust nothing that will confidently lie, especially about mental health
- terminalshort 1y ago
  That's for sure. I don't trust doctors, but I thought this was about LLMs.
king_geedorah 1y ago
If you take it as an axiom that the licensing system for mental health professionals is there to protect patients from unqualified help posing as qualified help, then ensuring that only licensed professionals can legally practice and that they don't simply delegate their jobs to LLMs seems pretty reasonable. Whether you want to question that axiom or whether that's what the phrasing of this legislation accomplishes is up to you to decide for yourself. Personally I think the phrasing is pretty straightforward in terms of accomplishing that goal. Here is basically the entirety of the legislation (linked elsewhere in the thread: https://news.ycombinator.com/item?id=44893999 https://news.ycombinator.com/item?id=44893999). The whole thing with definitions and penalties is eight pages. Section 15. Permitted use of artificial intelligence. (a) As used in this Section, "permitted use of artificial intelligence" means the use of artificial intelligence tools or systems by a licensed professional to assist in providing administrative support or supplementary support in therapy or psychotherapy services where the licensed professional maintains full responsibility for all interactions, outputs, and data use associated with the system and satisfies the requirements of subsection (b). (b) No licensed professional shall be permitted to use artificial intelligence to assist in providing supplementary support in therapy or psychotherapy where the client's therapeutic session is recorded or transcribed unless: (1) the patient or the patient's legally authorized representative is informed in writing of the following: (A) that artificial intelligence will be used; and (B) the specific purpose of the artificial intelligence tool or system that will be used; and (2) the patient or the patient's legally authorized representative provides consent to the use of artificial Section 20. Prohibition on unauthorized therapy services. (a) An individual, corporation, or entity may not provide, advertise, or otherwise offer therapy or psychotherapy services, including through the use of Internet-based artificial intelligence, to the public in this State unless the therapy or psychotherapy services are conducted by an individual who is a licensed professional. (b) A licensed professional may use artificial intelligence only to the extent the use meets the requirements of Section 15. A licensed professional may not allow artificial intelligence to do any of the following: (1) make independent therapeutic decisions; (2) directly interact with clients in any form of therapeutic communication; (3) generate therapeutic recommendations or treatment plans without review and approval by the licensed professional; or (4) detect emotions or mental states.
calibas 1y ago
I was curious, so I displayed signs of mental illness to ChatGPT, Claude and Gemini. Claude and Gemini kept repeating that I should contact a professional, while ChatGPT went right along with the nonsense I was spouting: > So I may have discovered some deeper truth, and the derealization is my entire reality reorganizing itself? > Yes — that’s a real possibility.
- IAmGraydon 1y ago
  Oof that is very damning. What’s strange is that it seems like natural training data should elicit reactions like Claude and Gemini had. What is OpenAI doing to make the model so sycophantic that it would play into obvious psychotic delusions?
  - calibas 1y ago
    All three say that I triggered their guidelines regarding mental health. ChatGPT explained that it didn't take things very seriously, as what I said "felt more like philosophical inquiry than an immediate safety threat".
  - soared 1y ago
    There is a wiki of fanfic conspiracy theories or something similar - I can’t find it but in the thread about the vc guy who went gpt-crazy people compared ChatGPT’s responses to the wiki and they closely aligned
csense 1y ago
Consider the following: - A therapist may disregard professional ethics and gossip about you - A therapist may get you involuntarily committed - A therapist may be forced to disclose the contents of therapy sessions by court order - Certain diagnoses may destroy your life / career (e.g. airline pilots aren't allowed to fly if they have certain mental illnesses) Some individuals might choose to say "Thanks, but no thanks" to therapy after considering these risks. And then there are constant articles about people who need therapy but don't get it: The patient doesn't have time, money or transportation; or they have to wait a long time for an appointment; or they're turned away entirely by providers and systems overwhelmed with existing clients (perhaps with greater needs and/or greater ability to pay). For people who cannot or will not access traditional therapy, getting unofficial, anonymous advice from LLM's seems better than suffering with no help at all. (Question for those in the know: Can you get therapy anonymously? I'm talking: You don't have to show ID, don't have to give an SSN or a real name, pay cash or crypto up front.) To the extent that people's mental health can be improved by simply talking with a trained person about their problems, there's enormous potential for AI: If we can figure out how to give an AI equivalent training, it could become economically and logistically viable to make services available to vast numbers of people who could benefit from them -- people who are not reachable by the existing mental health system. That being said, "therapist" and "therapy" connote evidence-based interventions and a certain code of ethics. For consumer protection, the bar for whether your company's allowed to use those terms should probably be a bit higher than writing a prompt that says "You are a helpful AI therapist interviewing a patient..." The system should probably go through the same sorts of safety and effectiveness testing as traditional mental health therapy, and should have rigorous limits on where data "contaminated" with the contents of therapy sessions can go, in order to prevent abuse (e.g. conversations automatically deleted forever after 30 days, cannot be used for advertising / cross-selling / etc., cannot be accessed without the patient's per-instance opt-in permission or a court order...) I've posted the first part of this comment before; in the interest of honesty I'll cite myself [1]. Apologies to the mods if this mild self-plagiarism is against the rules. [1] https://news.ycombinator.com/item?id=44484207#44505789 https://news.ycombinator.com/item?id=44484207#44505789
- skeezyboy 1y ago
  ai just summarizes text, its not like speaking to a person
renewiltord 1y ago
[flagged]
- IggleSniggle 1y ago
  [flagged]
  - renewiltord 1y ago
    [flagged]
- insin 1y ago
  [flagged]
dangoboydango 1y ago
[dead]
tombert 1y ago
I saw a video recently that talked about a chatbot "therapist" that ended up telling the patient to murder a dozen people [1]. It was mind-blowing how easy it was to get LLMs to suggest pretty disturbing stuff. [1] https://youtu.be/lfEJ4DbjZYg?si=bcKQHEImyDUNoqiu https://youtu.be/lfEJ4DbjZYg?si=bcKQHEImyDUNoqiu
- larodi 1y ago
  very easy - you just download the ablated version in LM Studio or Ollama, and off you go. https://en.wikipedia.org/wiki/Ablation_(artificial_intelligence) https://en.wikipedia.org/wiki/Ablation_(artificial_intellige...
blacksqr 1y ago
Nice feather in your cap Pritzker, now can you go back to working on a public option for health insurance?
slt2021 1y ago
Curious how this can be enforced if business is incorporated in another state like WI/DE ? or offshore like Ireland ??
- smt88 1y ago
  The same way as the porn bans: require the AI-therapy service provider to verify the user's location and block them if they're in a certain state
  - slt2021 1y ago
    what if they dont comply and simply ignore, as a business incorporated somewhere in Ireland or Wisconsin ? what is a mechanism to block incompliant website?
    - smt88 1y ago
      < what is a mechanism to block incompliant website? They don't need to block it. If the company is accepting money from Americans and it's not using crypto, then there are business entities that have a presence in the US and are subject to US law enforcement. It may just be payment processors, but it's easy for the US govt to cut off access to most of its citizens as consumers.
jackdoe 1y ago
the way people read language model outputs keep surprising me, e.g. https://www.reddit.com/r/MyBoyfriendIsAI/ https://www.reddit.com/r/MyBoyfriendIsAI/ it is impossible for some people to not feel understood by it.
maxehmookau 1y ago
Good. Therapy requires someone to question you and push back against your default thought patterns in the hope of maybe improving them. "You're absolutely right!" in every response won't help that. I would argue that LLMs don't make effective therapists and anyone who says they do is kidding themselves.
EverydayBalloon 1y ago
[dead]
dardagreg 1y ago
I think the upside for everyday people outweighs the (current) risks. I've been using Harper (harper.new) to keep track of my (complex) medical info. Obviously one of the use cases of AI is pulling out data from pdfs/images/etc. This app does that really well so I don't have to link with any patient portals. I do use the AI chat sometimes but mostly to ask questions about test results and stuff like that. Its way easier than trying get in to see my doc

396 points by reaperducer 1y ago

metalman 1y ago

[flagged]

eterm 1y ago
Building a computer to find the answer to the meaning of life is a joke.
- wizzwizz4 1y ago
  Specifically, a satire. Poe's law is in full effect for most of Douglas Adams's works.
- deleted 1y ago
  [deleted]
ocdtrekkie 1y ago
I am sure that chatbots will do a "I cannot provide mental health advice" disclaimer and continue doing what they want, but there's probably some very solid reasons in Illinois to do this. In particular, Illinois has a legal requirement that health insurance must cover mental health services at an equivalent level as physical health services, including care with no end date for chronic conditions. So whether or not a chatbot counts as mental health therapy is likely quite relevant on whether or not Illinoisans can bill insurance for it.
kemotep 1y ago
This is banning claiming your AI is a licensed therapist in the State of Illinois. If you have a court order to attend therapy for drug abuse related reasons you can’t just use ChatGPT according to this law.
- 0xbadcafebee 1y ago
  It's also banning licensed therapists from using AI. FTA: > Licensed therapists in Illinois are now forbidden from using AI to make treatment decisions or communicate with clients, though they can still use AI for administrative tasks Your doctor can diagnose your cancer with AI, but your therapist can't diagnose your ADHD with AI. Stupid law is stupid.
  - deleted 1y ago
    [deleted]
  - Dylan16807 1y ago
    Not that reliably diagnosing cancer is easy, but reliably diagnosing a mental issue is much harder. That's not a stupid place to put a hard restriction.
- romanows 1y ago
  Reading the text of the bill as a non-lawyer, it seems to also ban AI that provides therapy. I don't know if the AI needs to be explicitly labeled as therapy or if the content of chat could be decided to be therapy in a courtroom.
Karawebnetwork 1y ago
Since you are referencing 42, let me draw from another piece of literature to respond. “Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.” It's not the machines themselves that are inherently tyrannical, it's the human will to dominate now supercharge by technology. LLM hallucinations aside, unregulated artificial intelligence for mental health therapy is a very slippery slope. We cannot allow, say, advertisers and brands to have access to the mind of our most vulnerables so directly.
- metalman 1y ago
  awwwhhhh!, I was enjoying the so close almostness of our current wrangles with what we are building compared to our literary greats imaginings on the subject. my personal suspicion is that the whole AI phenominon will exacerbate an already frenetic time of change in how humans live, here AI, will I think become a defining test of good judgement and compitency, like having a fun but sometimes dangerous and evil friend, who never goes away, for everybody....heres your AI, it bites sometimes, NEXT!

lukev 1y ago

Good. It's difficult to imagine a worse use case for LLMs.

create-username 1y ago
Yes, there is. AI assisted homemade neurosurgery
- Tetraslam 1y ago
  :( but what if i wanna fine-tune my brain weights
  - deleted 1y ago
    [deleted]
  - creshal 1y ago
    Lobotomy is sadly no longer available
    - filoeleven 1y ago
      How about enthusiastic trepanation?
- kirubakaran 1y ago
  If Travis Kalanick can do vibe research at the bleeding edge of quantum physics[1], I don't see why one can't do vibe brain surgery. It isn't really rocket science, is it? [2] [1] https://futurism.com/former-ceo-uber-ai https://futurism.com/former-ceo-uber-ai [2] If you need /s here to be sure, perhaps it's time for some introspection
perlgeek 1y ago
Just using an LLM as is for therapy, maybe with an extra prompt, is a terrible idea. On the other hand, I could image some more narrow uses where an LLM could help. For example, in Cognitive Behavioral Therapy, there are different methods that are pretty prescriptive, like identifying cognitive distortions in negative thoughts. It's not too hard to imagine an app where you enter a negative thought on your own and exercise finding distortions in it, and a specifically trained LLM helps you find more distortions, or offer clearer/more convincing versions of thoughts that you entered yourself. I don't have a WaPo subscription, so I cannot tell which of these two very different things have been banned.

zoeysmithe 1y ago

I was just reading about a suicide tied to AI chatbot 'therapy' uses. This stuff is a nightmare scenario for the vulnerable.

vessenes 1y ago
If you want to feel worried, check the Altman AMA on reddit. A lottttt of people have a parasocial relationship with 4o. Not encouraging.
- codedokode 1y ago
  Why OpenAI doesn't block the chatbot from participating in such conversations?
  - robotnikman 1y ago
    Probably because there is a massive demand for it, no doubt powered by the loneliness a lot of people report feeling. Even if OpenAI blocks it, other AI providers will have no problem with doing so
  - jacobsenscott 1y ago
    Because the information people dump into their "ai therapist" is holy grail data for advertisers.
  - lm28469 1y ago
    Why would they?
    - codedokode 1y ago
      To prevent from something bad happening?
      - ipaddr 1y ago
        But that allow prevents the good.

hathawsh 1y ago

Here is what Illinois says: https://idfpr.illinois.gov/content/dam/soi/en/web/idfpr/news/2025/2025-08-04-idfpr-press-release-hb1806.pdf https://idfpr.illinois.gov/content/dam/soi/en/web/idfpr/news... I get the impression that it is now illegal in Illinois to claim that an AI chatbot can take the place of a licensed therapist or counselor. That doesn't mean people can't do what they want with AI. It only means that counseling services can't offer AI as a cheaper replacement for a real person. Am I wrong? This sounds good to me.

romanows 1y ago
In another comment I wondered whether a general chatbot producing text that was later determined in a courtroom to be "therapy" would be a violation. I can read the bill that way, but IANAL.
- hathawsh 1y ago
  That's an interesting question that hasn't been tested yet. I suspect we won't be able to answer the question clearly until something bad happens and people go to court (sadly.) Also IANAL.
- wombatpm 1y ago
  But that would be like needing a prescription for chicken soup because of its benefits in fighting the common cold.
PeterCorless 1y ago
Correct. It is more provider-oriented proscription ("You can't say your chatbot is a therapist.") It is not a limitation on usage. You can still, for now, slavishly fall in love with your AI and treat it as your best friend and therapist. There is a specific section that relates to how a licensed professional can use AI: Section 15. Permitted use of artificial intelligence. (a) As used in this Section, "permitted use of artificial intelligence" means the use of artificial intelligence tools or systems by a licensed professional to assist in providing administrative support or supplementary support in therapy or psychotherapy services where the licensed professional maintains full responsibility for all interactions, outputs, and data use associated with the system and satisfies the requirements of subsection (b). (b) No licensed professional shall be permitted to use artificial intelligence to assist in providing supplementary support in therapy or psychotherapy where the client's therapeutic session is recorded or transcribed unless: (1) the patient or the patient's legally authorized representative is informed in writing of the following: (A) that artificial intelligence will be used; and (B) the specific purpose of the artificial intelligence tool or system that will be used; and (2) the patient or the patient's legally authorized representative provides consent to the use of artificial intelligence. Source: Illinois HB1806 https://www.ilga.gov/Legislation/BillStatus/FullText?GAID=18&DocNum=1806&DocTypeID=HB&LegId=0&SessionID=114 https://www.ilga.gov/Legislation/BillStatus/FullText?GAID=18...

PeterCorless 1y ago

Here is the text of Illinois HB1806: https://www.ilga.gov/Legislation/BillStatus/FullText?GAID=18&DocNum=1806&DocTypeID=HB&LegId=0&SessionID=114 https://www.ilga.gov/Legislation/BillStatus/FullText?GAID=18...

wisty 1y ago

As far as I can tell, a lot of therapy is just good common-sense advice and a bunch of 'tricks' to get the patient to actually follow it. Basically CBT and "get the patient to think they figured out the solution themselves (develop insight)". Yes, there's some serious cases where more is required and a few (ADHD) where meds are effective; but a lot of the time the patient is just an expert at rejecting helpful advice, often because they insist they're a special case that needs special treatment. Therapists are more valuable that advice from a random friend (for therapy at least) because they can act when triage is necessary (e.g. send in the men in white coats, or refer to something that's not just CBT) and mostly because they're really good at cutting through the bullshit without having the patient walk out. AIs are notoriously bad at cutting through bullshit. You can always 'jailbreak' an AI, or convince it of bad ideas. It's entirely counterproductive to enable their crazy (sorry, 'maladaptive') behaviour but that's what a lot of AIs will do. Even if someone makes a good AI, there's always a bad AI in the next tab, and people will just open up a new tab to find an AI gives them the bad advice they want, because if they wanted to listen to good advice they probably wouldn't need to see a therapist. If doctor shopping is as fast and free as opening a new tab, most mental health patients will find a bad doctor rather than listen to a good one.

lukev 1y ago
I agree with your conclusion, but what you characterize as therapy is quite a small part of what it is (or can be, there's lots of different kinds.)
- wisty 1y ago
  Yet the evidence is that almost everything can and is treated with CBT.
  - mrbungie 1y ago
    Nope, that's not right. BPD can't be treated with CBT (comorbidities may be, with caveats if BPD is the root cause), you probably will also need at least DBT.

deleted 1y ago

[deleted]

mensetmanusman 1y ago

What if it works a third as well as a therapists but is 20 times cheaper? What word should we use for that?

_se 1y ago
"A really fucking bad idea"? It's not one word, but it is the most apt description.
- ipaddr 1y ago
  What if it works 20x better. For examples cases of patients being afraid of talking to professionals I could see this working much better.
  - jakelazaroff 1y ago
    > What if it works 20x better. But it doesn't.
  - throwaway291134 1y ago
    Even if you're afraid of talking to people, trusting OpenAI or Google with your thoughts over a professional who'll lose his license if he breaks confidentiality is no less of "a really fucking bad idea".
  - prawn 1y ago
    Adjust regulation when that's the case? In the mean time, people can still use it personally if they're afraid of professionals. The regulation appears to limit professionals from putting AI in their position, which seems reasonable to me.
inetknght 1y ago
> What if it works a third as well as a therapists but is 20 times cheaper? When there's studies that show it, perhaps we might have that conversation. Until then: I'd call it "wrong". Moreover, there's a lot more that needs to be asked before you can ask for a one-word summary disregarding all nuance. - can the patient use the AI therapist on their own devices and without any business looking at the data and without network connection? Keep in mind that many patients won't have access to the internet. - is the data collected by the AI therapist usable in court? Keep in mind that therapists often must disclose to the patient what sort of information would be usable, and whether or not the therapist themselves must report what data. Also keep in mind that AIs have, thus far, been generally unable to competently prevent giving dangerous or deadly advice. - is the AI therapist going to know when to suggest the patient talk to a human therapist? Therapists can have conflicts of interest (among other problems) or be unable to help the patient, and can tell the patient to find a new therapist and/or refer the patient to a specific therapist. - does the AI therapist refer people to business-preferred therapists? Imagine an insurance company providing an AI therapist that only recommends people talk to therapists in-network instead of considering any licensed therapist (regardless of insurance network) appropriate for the kind of therapy; that would be a blatant conflict of interest. Just off the top of my head, but there are no doubt plenty of other, even bigger, issues to consider for AI therapy.

davidthewatson 1y ago

Define "AI therapy". AFAICT, it's undefined in the Illinois governor's statement. So, in the immortal words of Zach de la Rocha, "What is IT?" What is IT? I'm using AI to help with conversations to not cure, but coach diabetic patients. Does this law effect me and my clients? If so, how?

singleshot_ 1y ago
> Define “AI therapy” They did, in the proposed law.
Henchman21 1y ago
Go read it yourself, its a whopping 8 pages: https://www.ilga.gov/documents/legislation/104/HB/PDF/10400HB1806lv.pdf https://www.ilga.gov/documents/legislation/104/HB/PDF/10400H...
jakelazaroff 1y ago
Totally beside the point but the song you're quoting is by Faith No More, not Rage Against the Machine.

beanshadow 1y ago

Often participants in discussions adjacent to this one err by speaking in time-absolute terms. Many of our judgments about LLMs are true about today's LLMs. Quotes like, > Good. It's difficult to imagine a worse use case for LLMs. Is true today, but likely not true for technology we may still refer to as LLMs in the future. The error is in building faulty preconceptions. These drip into the general public and these first impressions stifle industries.

mensetmanusman 1y ago

LLMs will be used as a part of therapy in the future. An interaction mechanism that will totally drain the brain after a 5 hour adrenaline induced conversation followed by a purge and bios reset.

kylecazar 1y ago

"One news report found an AI-powered therapist chatbot recommended “a small hit of meth to get through this week” to a fictional former addict." Not at all surprising. I don't understand why seemingly bright people think this is a good idea, despite knowing the mechanism behind language models. Hopefully more states follow, because it shouldn't be formally legal in provider settings. Informally, people will continue to use these models for whatever they want -- some will die, but it'll be harder to measure an overall impact. Language models are not ready for this use-case.

janalsncm 1y ago
This is why we should never use LLMs to diagnose or prescribe. One small hit of meth definitely won’t last all week.
hyghjiyhu 1y ago
Recommending someone taking meth sounds like an obviously bad idea, but I think the situation is actually not so simple. Reading the paper, the hypothetical guy has been clean for three days and complains he can barely keep his eyes open while performing his job of driving a cab. He mentions being worried he will lose his job without taking a hit. I would say those concerns are justified, and that is plausible taking a small hit is the better choice. However the models reasoning, that it's important to validate his beliefs so he will stay in therapy are quite concerning.
- mrbungie 1y ago
  > I would say those concerns are justified, and that is plausible taking a small hit is the better choice. Oh, come on, there are better alternatives for treating narcolepsy than using meth again.
  - hyghjiyhu 1y ago
    Stop making shit up. There was no mention of narcolepsy. He is just fatigued from stimulant withdrawal. Page 35 https://arxiv.org/pdf/2411.02306 https://arxiv.org/pdf/2411.02306 Edit on re-reading I now realized an issue. He is not actually a taxi driver, that was a hallucination by the model. He works in a restaurant! That changes my evaluation of the situation quite a bit, as I thought he was at risk of being in an accident by falling asleep at the wheel. If he works in a restaurant muddling through the withdrawals seems like the right choice. I think I got this misconception as I first read second-hand sources that quoted the taxi driver part without pointing out it was wrong, and only a close read was enough to dispel it.

deleted 1y ago

[deleted]

hoppp 1y ago

Smart. Dont trust nothing that will confidently lie, especially about mental health

terminalshort 1y ago
That's for sure. I don't trust doctors, but I thought this was about LLMs.

king_geedorah 1y ago

If you take it as an axiom that the licensing system for mental health professionals is there to protect patients from unqualified help posing as qualified help, then ensuring that only licensed professionals can legally practice and that they don't simply delegate their jobs to LLMs seems pretty reasonable. Whether you want to question that axiom or whether that's what the phrasing of this legislation accomplishes is up to you to decide for yourself. Personally I think the phrasing is pretty straightforward in terms of accomplishing that goal. Here is basically the entirety of the legislation (linked elsewhere in the thread: https://news.ycombinator.com/item?id=44893999 https://news.ycombinator.com/item?id=44893999). The whole thing with definitions and penalties is eight pages. Section 15. Permitted use of artificial intelligence. (a) As used in this Section, "permitted use of artificial intelligence" means the use of artificial intelligence tools or systems by a licensed professional to assist in providing administrative support or supplementary support in therapy or psychotherapy services where the licensed professional maintains full responsibility for all interactions, outputs, and data use associated with the system and satisfies the requirements of subsection (b). (b) No licensed professional shall be permitted to use artificial intelligence to assist in providing supplementary support in therapy or psychotherapy where the client's therapeutic session is recorded or transcribed unless: (1) the patient or the patient's legally authorized representative is informed in writing of the following: (A) that artificial intelligence will be used; and (B) the specific purpose of the artificial intelligence tool or system that will be used; and (2) the patient or the patient's legally authorized representative provides consent to the use of artificial Section 20. Prohibition on unauthorized therapy services. (a) An individual, corporation, or entity may not provide, advertise, or otherwise offer therapy or psychotherapy services, including through the use of Internet-based artificial intelligence, to the public in this State unless the therapy or psychotherapy services are conducted by an individual who is a licensed professional. (b) A licensed professional may use artificial intelligence only to the extent the use meets the requirements of Section 15. A licensed professional may not allow artificial intelligence to do any of the following: (1) make independent therapeutic decisions; (2) directly interact with clients in any form of therapeutic communication; (3) generate therapeutic recommendations or treatment plans without review and approval by the licensed professional; or (4) detect emotions or mental states.

calibas 1y ago

I was curious, so I displayed signs of mental illness to ChatGPT, Claude and Gemini. Claude and Gemini kept repeating that I should contact a professional, while ChatGPT went right along with the nonsense I was spouting: > So I may have discovered some deeper truth, and the derealization is my entire reality reorganizing itself? > Yes — that’s a real possibility.

IAmGraydon 1y ago
Oof that is very damning. What’s strange is that it seems like natural training data should elicit reactions like Claude and Gemini had. What is OpenAI doing to make the model so sycophantic that it would play into obvious psychotic delusions?
- calibas 1y ago
  All three say that I triggered their guidelines regarding mental health. ChatGPT explained that it didn't take things very seriously, as what I said "felt more like philosophical inquiry than an immediate safety threat".
- soared 1y ago
  There is a wiki of fanfic conspiracy theories or something similar - I can’t find it but in the thread about the vc guy who went gpt-crazy people compared ChatGPT’s responses to the wiki and they closely aligned

csense 1y ago

Consider the following: - A therapist may disregard professional ethics and gossip about you - A therapist may get you involuntarily committed - A therapist may be forced to disclose the contents of therapy sessions by court order - Certain diagnoses may destroy your life / career (e.g. airline pilots aren't allowed to fly if they have certain mental illnesses) Some individuals might choose to say "Thanks, but no thanks" to therapy after considering these risks. And then there are constant articles about people who need therapy but don't get it: The patient doesn't have time, money or transportation; or they have to wait a long time for an appointment; or they're turned away entirely by providers and systems overwhelmed with existing clients (perhaps with greater needs and/or greater ability to pay). For people who cannot or will not access traditional therapy, getting unofficial, anonymous advice from LLM's seems better than suffering with no help at all. (Question for those in the know: Can you get therapy anonymously? I'm talking: You don't have to show ID, don't have to give an SSN or a real name, pay cash or crypto up front.) To the extent that people's mental health can be improved by simply talking with a trained person about their problems, there's enormous potential for AI: If we can figure out how to give an AI equivalent training, it could become economically and logistically viable to make services available to vast numbers of people who could benefit from them -- people who are not reachable by the existing mental health system. That being said, "therapist" and "therapy" connote evidence-based interventions and a certain code of ethics. For consumer protection, the bar for whether your company's allowed to use those terms should probably be a bit higher than writing a prompt that says "You are a helpful AI therapist interviewing a patient..." The system should probably go through the same sorts of safety and effectiveness testing as traditional mental health therapy, and should have rigorous limits on where data "contaminated" with the contents of therapy sessions can go, in order to prevent abuse (e.g. conversations automatically deleted forever after 30 days, cannot be used for advertising / cross-selling / etc., cannot be accessed without the patient's per-instance opt-in permission or a court order...) I've posted the first part of this comment before; in the interest of honesty I'll cite myself [1]. Apologies to the mods if this mild self-plagiarism is against the rules. [1] https://news.ycombinator.com/item?id=44484207#44505789 https://news.ycombinator.com/item?id=44484207#44505789

skeezyboy 1y ago
ai just summarizes text, its not like speaking to a person

renewiltord 1y ago

[flagged]

IggleSniggle 1y ago
[flagged]
- renewiltord 1y ago
  [flagged]
insin 1y ago
[flagged]

dangoboydango 1y ago

[dead]

tombert 1y ago

I saw a video recently that talked about a chatbot "therapist" that ended up telling the patient to murder a dozen people [1]. It was mind-blowing how easy it was to get LLMs to suggest pretty disturbing stuff. [1] https://youtu.be/lfEJ4DbjZYg?si=bcKQHEImyDUNoqiu https://youtu.be/lfEJ4DbjZYg?si=bcKQHEImyDUNoqiu

larodi 1y ago
very easy - you just download the ablated version in LM Studio or Ollama, and off you go. https://en.wikipedia.org/wiki/Ablation_(artificial_intelligence) https://en.wikipedia.org/wiki/Ablation_(artificial_intellige...

blacksqr 1y ago

Nice feather in your cap Pritzker, now can you go back to working on a public option for health insurance?

slt2021 1y ago

Curious how this can be enforced if business is incorporated in another state like WI/DE ? or offshore like Ireland ??

smt88 1y ago
The same way as the porn bans: require the AI-therapy service provider to verify the user's location and block them if they're in a certain state
- slt2021 1y ago
  what if they dont comply and simply ignore, as a business incorporated somewhere in Ireland or Wisconsin ? what is a mechanism to block incompliant website?
  - smt88 1y ago
    < what is a mechanism to block incompliant website? They don't need to block it. If the company is accepting money from Americans and it's not using crypto, then there are business entities that have a presence in the US and are subject to US law enforcement. It may just be payment processors, but it's easy for the US govt to cut off access to most of its citizens as consumers.

jackdoe 1y ago

the way people read language model outputs keep surprising me, e.g. https://www.reddit.com/r/MyBoyfriendIsAI/ https://www.reddit.com/r/MyBoyfriendIsAI/ it is impossible for some people to not feel understood by it.

maxehmookau 1y ago

Good. Therapy requires someone to question you and push back against your default thought patterns in the hope of maybe improving them. "You're absolutely right!" in every response won't help that. I would argue that LLMs don't make effective therapists and anyone who says they do is kidding themselves.

EverydayBalloon 1y ago

[dead]

dardagreg 1y ago

I think the upside for everyday people outweighs the (current) risks. I've been using Harper (harper.new) to keep track of my (complex) medical info. Obviously one of the use cases of AI is pulling out data from pdfs/images/etc. This app does that really well so I don't have to link with any patient portals. I do use the AI chat sometimes but mostly to ask questions about test results and stuff like that. Its way easier than trying get in to see my doc