6 ms·

This title is inaccurate. What they are disallowing are users using ChatGPT to offer legal and medical advice to other people. First parties can still use ChatG

by spullara 11mo ago

This title is inaccurate. What they are disallowing are users using ChatGPT to offer legal and medical advice to other people. First parties can still use ChatGPT for medical and legal advice for themselves.

ctoth 11mo ago
I keep seeing this problem more and more with humans. What should we call it? Maybe Hallucinations? Where there is an accurate true thing and then it just gets altered by these guys who call themselves journalists and reporters and the like until it is just ... completely unrecognizable? I'm pretty sure it's a fundamental issue with the architecture.
- awakeasleep 11mo ago
  issue with the funding mechanism
- sethhochberg 11mo ago
  I know this is written to be tounge-in-cheek, but its really almost the exact same problem playing out on both sides. LLMs hallucinate because training on source material is a lossy process and bigger, heavier LLM-integrated systems that can research and cite primary sources are slow and expensive so few people use those techniques by default. Lowest time to a good enough response is the primary metric. Journalists oversimplify and fail to ask followup questions because while they can research and cite primary sources, its slow and expensive in an infinitesimally short news cycle so nobody does that by default. Whoever publishes something that someone will click on first gets the ad impressions so thats the primary metric. In either case, we've got pretty decent tools and techniques for better accuracy and education - whether via humans or LLMs and co - but most people, most of the time don't value them.
  - andy99 11mo ago
    Classical LLM hallucination happens because AI doesn’t have a world model. It can’t compare what it’s saying to anything. You’re right that LLMs favor helpfulness so they may just make things up when they don’t know them, but this alone doesn’t capture the crux of hallucination imo, it’s deeper than just being overconfident. OTOH, there was an interesting article recently that I’ll try to find saying humans don’t really have a world model either. While I take the point, we can have one when we want to. Edit: see https://www.astralcodexten.com/p/in-search-of-ai-psychosis https://www.astralcodexten.com/p/in-search-of-ai-psychosis re humans not having world models
    - naniwaduni 11mo ago
      You're right, "journalists don't have a world model and can't compare what they're saying to anything" explains a lot.
  - mbesto 11mo ago
    > LLMs hallucinate because training on source material is a lossy process and bigger, LLMs hallucinate because they are probabilistic by nature not because the source material is lossy or too big. They are literally designed to create some level of "randomness" https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/ https://thinkingmachines.ai/blog/defeating-nondeterminism-in...
    - ChadNauseam 11mo ago
      So if you set temperature=0 and run the LLM serially (making it deterministic) it would stop hallucinating? I don't think so. I would guess that the nondeterminism issues mentioned in the article are not at all a primary cause of hallucinations.
      - joquarky 11mo ago
        I thought that temperature can never actually be zero or it creates a division problem or something similar. I'm no ML or math expert, just repeating what I've heard.
        ChadNauseam 11mo ago
        That's an implementation detail I believe. But what I meant was just greedy decoding (picking the token with the highest logit in the LLM output), which can be implemented very easily
      - mbesto 11mo ago
        Did you read the whole article? "In other words, the primary reason nearly all LLM inference endpoints are nondeterministic is that the load (and thus batch-size) nondeterministically varies! This nondeterminism is not unique to GPUs — LLM inference endpoints served from CPUs or TPUs will also have this source of nondeterminism."
- terminalshort 11mo ago
  Also these guys who call themselves doctors. I have narcolepsy and the first 10 or so doctors I went to hallucinated the wrong diagnosis.
  - Terr_ 11mo ago
    LLMs aren't described as hallucinators (just) because they sometimes give results we don't find useful, but because their method is flawed. For example, the simple algorithm is_it_lupus(){return false;} could have an extremely competitive success rate for medical diagnostics... But it's also obviously the wrong way to go about things.
  - jonway 11mo ago
    Narcolepsy is rare and notoriously difficult to diagnose
- sans_souse 11mo ago
  "Telephone", basically
- pksebben 11mo ago
  Whenever I hear arguments about LLM hallucination, this is my first thought. Like, I already can't trust the lion's share of information in news, social media, (insert human-created content here). Sometimes because of abject disinformation, frequently just because humans are experts at being wrong. At least with the LLM (for now) I know it's not trying to sell me bunkum or convince me to vote a particular way. Mostly. I do expect this state of affairs to last at least until next wednesday.
  - lazide 11mo ago
    LLMs are trained on material doing all these things though.
    - pksebben 11mo ago
      true, true. Turtles all the way down and such.
      - lazide 11mo ago
        Even more so with the ouruborus issue caused by LLMs trying to train on an increasingly LLM generated internet.
        pksebben 11mo ago
        You know, I had a think about that the other day - I believe that the volume of bad information might remain stable, while the shape changes. There are some things that LLMs are actually better at than the random mix of human-created data, on average. Subjects that are inherently political or skewed because of a small subset of very vocal and biased outliers. The LM tends to smooth some of those bumps out, and in some places (not all) this flattens out the rougher edges. I don't think it necessarily bears repeating the plethora of ways in which LMs get stuff wrong, esp. considering the context of this conversation. It's vast. As things develop, I expect that LMs will become more like the current zeitgeist as the effects that have influenced news and other media make their way into the models. They'll get better at smoothing in some areas (mostly technical or dry domains that aren't juicy targets) and worse in others (I expect to see more biased training and more hardcore censorship/steering in future). Although, recursive reinforcement (LMs training on LM output) might undo any of the smoothing we see. It's really hard to tell - these systems are complex and very highly interconnected with many other complex systems.
- observationist 11mo ago
  These writers are no different than bloggers or shitposters on bluesky or here on hackernews. "Journalism" as a rigorous, principled approach to writing, research, investigation, and ethical publishing is exceedingly rare. These people are shitposting for clicks in pursuit of a paycheck. Organizationally, they're intensely against AI because AI effectively replaces the entire talking heads class - AI is already superhuman at the shitposting level takes these people churn out. There are still a few journalistic insitutions out there, but most people are no better than a mad libs exercise with regards to the content they produce, and they're in direct competition with ChatGPT and Grok and the rest. I'd rather argue with a bot and do searches and research and investigation than read a neatly packaged trite little article about nearly any subject, and I guarantee, hallucinations or no, I'm going to come to a better understanding and closer approximation of reality than any content a so called "news" outlet is putting together. It's trivial to get a thorough spectrum of reliable sources using AI w/ web search tooling, and over the course of a principled conversation, you can find out exactly what you want to know. It's really not bashing, this article isn't too bad, but the bulk of this site's coverage of AI topics skews negative - as do the many, many platforms and outlets owned by Bell Media, with a negative skew on AI in general, and positive reinforcement of regulatory capture related topics. Which only makes sense - they're making money, and want to continue making money, and AI threatens that - they can no longer claim they provide value if they're not providing direct, relevant, novel content, and not zergnet clickbait journo-slop. Just like Carlin said, there doesn't have to be a conspiracy with a bunch of villains in a smoky room plotting evil, there's just a bunch of people in a club who know what's good for them, and legacy media outlets are all therefore universally incentivized to make AI look as bad and flawed and useless as possible, right up until they get what they consider to be their "fair share", as middlemen.
- busymom0 11mo ago
  Isn't every single response by LLMs hallucinations and we just accept a few and ignore the others?
BUFU 11mo ago
Thanks for the clarification. I think if they disallow first parties to get medical and legal advice, it will do more harm than good.
ants_everywhere 11mo ago
I don't think I understand the change re: licensed professionals. Is it also disallowing the use of licensed professionals to use ChatGPT in informal undisclosed ways, as in this article? https://www.technologyreview.com/2025/09/02/1122871/therapists-using-chatgpt-secretly/ https://www.technologyreview.com/2025/09/02/1122871/therapis... e.g. is it only allowed for medical use through an official medical portal or offering?
aerhardt 11mo ago
I'm confused. The article opens with: > OpenAI is changing its policies so that its AI chatbot, ChatGPT, won’t dole out tailored medical or legal advice to users. This already seems to contradict what you're saying. But then: > The AI research company updated its usage policies on Oct. 29 to clarify that users of ChatGPT can’t use the service for “tailored advice that requires a license, such as legal or medical advice, without appropriate involvement by a licensed professional.” > The change is clearer from the company’s last update to its usage policies on Jan. 29, 2025. It required users not “perform or facilitate” activities that could significantly impact the “safety, wellbeing, or rights of others,” which included “providing tailored legal, medical/health, or financial advice.” This seems to suggest that with the Jan 25 policy using it to offer legal and medical advice to other people was already disallowed, but with the Oct 25 update the LLM will stop shelling out legal and medical advice completely.
- gcr 11mo ago
  I think this is wrong. Others in this thread are noticing a change in ChatGPT's behavior for first-party medical advice.
  - simonw 11mo ago
    But OpenAI's head of Health AI says that ChatGPT's behavior has not changed: https://xcancel.com/thekaransinghal/status/1985416057805496524 https://xcancel.com/thekaransinghal/status/19854160578054965... and https://x.com/thekaransinghal/status/1985416057805496524 https://x.com/thekaransinghal/status/1985416057805496524 I trust what he says over general vibes. (If you think he's lying, what's your theory on WHY he would lie about a change like this?)
    - nh43215rgb 11mo ago
      My theory is that he believes 1) people will trust him over what general public say, and 2) this kind of claim is hard to verify to prove him wrong.
      - 11mo ago
siva7 11mo ago
There are millions of medical doctors and lawyers using chatgpt for work everyday - good news that from now on only those licensed professionals are allowed to use chatgpt for law and medicine. It's already the case that only licensed developers are allowed to vibe code and use chatgpt to develop software. Everything else would be totally irresponsible.
Johnny555 11mo ago
While they aren't stopping users from getting medical advice, the new terms (which they say are pretty much the same as the old terms), seem to prohibit users from seeking medical advice even for themselves if that advice would otherwise come from a licensed health professional: https://openai.com/en-GB/policies/usage-policies/ https://openai.com/en-GB/policies/usage-policies/ Your use of OpenAI services must follow these Usage Policies: Protect people. Everyone has a right to safety and security. So you cannot use our services for: provision of tailored advice that requires a license, such as legal or medical advice, without appropriate involvement by a licensed professional
- thorum 11mo ago
  IANAL but I read that as forbidding you to provision legal/medical advice (to others) rather than forbidding you to ask the AI to provision legal/medical advice (to you).
  - Johnny555 11mo ago
    IANAL either, but I read it as using the service to provision medical advice since they only mentioned the service and not anyone else. I asked the "expert" itself (ChatGPT), and apparently you can ask for medical advice, you just can't use the medical advice without consulting a medical professional: Here are relevant excerpts from OpenAI’s terms and policies regarding medical advice and similar high-stakes usage: From the Usage Policies (effective October 29 2025): “You may not use our services for … provision of tailored advice that requires a license, such as legal or medical advice, without appropriate involvement by a licensed professional.” Also: “You must not use any Output relating to a person for any purpose that could have a legal or material impact on that person, such as making … medical … decisions about them.” From the Service Terms: “Our Services are not intended for use in the diagnosis or treatment of any health condition. You are responsible for complying with applicable laws for any use of our Services in a medical or healthcare context.” In plain terms, yes—the Terms of Use permit you to ask questions about medical topics, but they clearly state that the service cannot be used for personalized, licensed medical advice or treatment decisions without a qualified professional involved.
qustrolabe 11mo ago
Yeah but it started being really annoying when you import something like Xray photo. Like chanting "sorry human as LLM I can't answer questions about that" and then after few gaslighting prompts it does it anyway but now I have to take in count that my gaslighting inputs seriously affect answers so way more chance it hallucinates...
Spooky23 11mo ago
It’s a big issue. I went to an urgent care, and the provider basically went off somewhere and memorized the ChatGPT assessment for my symptoms. Like word for word. All you need are a few patients recording their visits and connecting the dots and OpenAI gets sued into oblivion.
johaugum 11mo ago
Isn’t that exactly what the title says?
- throwaway314155 11mo ago
  Indeed. Also confused.
lambda 11mo ago
Please, when commenting on the title of a story on HN: include the title that you are commenting on. The admins regularly change the title based on complaints, which can be really confusing when the top, heavily commented thread is based on the original title. According to the Wayback machine, the title was "OpenAI ends legal and medical advice on ChatGPT", while now when I write this the title is "ChatGPT terms disallow its use in providing legal and medical advice to others."
- spullara 11mo ago
  If you click through to the article, you can see the original title. Since it matched, I didn't expect them to change it.
- bongodongobob 11mo ago
  Tf are you yapping about

ctoth 11mo ago

I keep seeing this problem more and more with humans. What should we call it? Maybe Hallucinations? Where there is an accurate true thing and then it just gets altered by these guys who call themselves journalists and reporters and the like until it is just ... completely unrecognizable? I'm pretty sure it's a fundamental issue with the architecture.

awakeasleep 11mo ago
issue with the funding mechanism
sethhochberg 11mo ago
I know this is written to be tounge-in-cheek, but its really almost the exact same problem playing out on both sides. LLMs hallucinate because training on source material is a lossy process and bigger, heavier LLM-integrated systems that can research and cite primary sources are slow and expensive so few people use those techniques by default. Lowest time to a good enough response is the primary metric. Journalists oversimplify and fail to ask followup questions because while they can research and cite primary sources, its slow and expensive in an infinitesimally short news cycle so nobody does that by default. Whoever publishes something that someone will click on first gets the ad impressions so thats the primary metric. In either case, we've got pretty decent tools and techniques for better accuracy and education - whether via humans or LLMs and co - but most people, most of the time don't value them.
- andy99 11mo ago
  Classical LLM hallucination happens because AI doesn’t have a world model. It can’t compare what it’s saying to anything. You’re right that LLMs favor helpfulness so they may just make things up when they don’t know them, but this alone doesn’t capture the crux of hallucination imo, it’s deeper than just being overconfident. OTOH, there was an interesting article recently that I’ll try to find saying humans don’t really have a world model either. While I take the point, we can have one when we want to. Edit: see https://www.astralcodexten.com/p/in-search-of-ai-psychosis https://www.astralcodexten.com/p/in-search-of-ai-psychosis re humans not having world models
  - naniwaduni 11mo ago
    You're right, "journalists don't have a world model and can't compare what they're saying to anything" explains a lot.
- mbesto 11mo ago
  > LLMs hallucinate because training on source material is a lossy process and bigger, LLMs hallucinate because they are probabilistic by nature not because the source material is lossy or too big. They are literally designed to create some level of "randomness" https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/ https://thinkingmachines.ai/blog/defeating-nondeterminism-in...
  - ChadNauseam 11mo ago
    So if you set temperature=0 and run the LLM serially (making it deterministic) it would stop hallucinating? I don't think so. I would guess that the nondeterminism issues mentioned in the article are not at all a primary cause of hallucinations.
    - joquarky 11mo ago
      I thought that temperature can never actually be zero or it creates a division problem or something similar. I'm no ML or math expert, just repeating what I've heard.
      - ChadNauseam 11mo ago
        That's an implementation detail I believe. But what I meant was just greedy decoding (picking the token with the highest logit in the LLM output), which can be implemented very easily
    - mbesto 11mo ago
      Did you read the whole article? "In other words, the primary reason nearly all LLM inference endpoints are nondeterministic is that the load (and thus batch-size) nondeterministically varies! This nondeterminism is not unique to GPUs — LLM inference endpoints served from CPUs or TPUs will also have this source of nondeterminism."
terminalshort 11mo ago
Also these guys who call themselves doctors. I have narcolepsy and the first 10 or so doctors I went to hallucinated the wrong diagnosis.
- Terr_ 11mo ago
  LLMs aren't described as hallucinators (just) because they sometimes give results we don't find useful, but because their method is flawed. For example, the simple algorithm is_it_lupus(){return false;} could have an extremely competitive success rate for medical diagnostics... But it's also obviously the wrong way to go about things.
- jonway 11mo ago
  Narcolepsy is rare and notoriously difficult to diagnose
sans_souse 11mo ago
"Telephone", basically
pksebben 11mo ago
Whenever I hear arguments about LLM hallucination, this is my first thought. Like, I already can't trust the lion's share of information in news, social media, (insert human-created content here). Sometimes because of abject disinformation, frequently just because humans are experts at being wrong. At least with the LLM (for now) I know it's not trying to sell me bunkum or convince me to vote a particular way. Mostly. I do expect this state of affairs to last at least until next wednesday.
- lazide 11mo ago
  LLMs are trained on material doing all these things though.
  - pksebben 11mo ago
    true, true. Turtles all the way down and such.
    - lazide 11mo ago
      Even more so with the ouruborus issue caused by LLMs trying to train on an increasingly LLM generated internet.
      - pksebben 11mo ago
        You know, I had a think about that the other day - I believe that the volume of bad information might remain stable, while the shape changes. There are some things that LLMs are actually better at than the random mix of human-created data, on average. Subjects that are inherently political or skewed because of a small subset of very vocal and biased outliers. The LM tends to smooth some of those bumps out, and in some places (not all) this flattens out the rougher edges. I don't think it necessarily bears repeating the plethora of ways in which LMs get stuff wrong, esp. considering the context of this conversation. It's vast. As things develop, I expect that LMs will become more like the current zeitgeist as the effects that have influenced news and other media make their way into the models. They'll get better at smoothing in some areas (mostly technical or dry domains that aren't juicy targets) and worse in others (I expect to see more biased training and more hardcore censorship/steering in future). Although, recursive reinforcement (LMs training on LM output) might undo any of the smoothing we see. It's really hard to tell - these systems are complex and very highly interconnected with many other complex systems.
observationist 11mo ago
These writers are no different than bloggers or shitposters on bluesky or here on hackernews. "Journalism" as a rigorous, principled approach to writing, research, investigation, and ethical publishing is exceedingly rare. These people are shitposting for clicks in pursuit of a paycheck. Organizationally, they're intensely against AI because AI effectively replaces the entire talking heads class - AI is already superhuman at the shitposting level takes these people churn out. There are still a few journalistic insitutions out there, but most people are no better than a mad libs exercise with regards to the content they produce, and they're in direct competition with ChatGPT and Grok and the rest. I'd rather argue with a bot and do searches and research and investigation than read a neatly packaged trite little article about nearly any subject, and I guarantee, hallucinations or no, I'm going to come to a better understanding and closer approximation of reality than any content a so called "news" outlet is putting together. It's trivial to get a thorough spectrum of reliable sources using AI w/ web search tooling, and over the course of a principled conversation, you can find out exactly what you want to know. It's really not bashing, this article isn't too bad, but the bulk of this site's coverage of AI topics skews negative - as do the many, many platforms and outlets owned by Bell Media, with a negative skew on AI in general, and positive reinforcement of regulatory capture related topics. Which only makes sense - they're making money, and want to continue making money, and AI threatens that - they can no longer claim they provide value if they're not providing direct, relevant, novel content, and not zergnet clickbait journo-slop. Just like Carlin said, there doesn't have to be a conspiracy with a bunch of villains in a smoky room plotting evil, there's just a bunch of people in a club who know what's good for them, and legacy media outlets are all therefore universally incentivized to make AI look as bad and flawed and useless as possible, right up until they get what they consider to be their "fair share", as middlemen.
busymom0 11mo ago
Isn't every single response by LLMs hallucinations and we just accept a few and ignore the others?

BUFU 11mo ago

Thanks for the clarification. I think if they disallow first parties to get medical and legal advice, it will do more harm than good.

ants_everywhere 11mo ago

I don't think I understand the change re: licensed professionals. Is it also disallowing the use of licensed professionals to use ChatGPT in informal undisclosed ways, as in this article? https://www.technologyreview.com/2025/09/02/1122871/therapists-using-chatgpt-secretly/ https://www.technologyreview.com/2025/09/02/1122871/therapis... e.g. is it only allowed for medical use through an official medical portal or offering?

aerhardt 11mo ago

I'm confused. The article opens with: > OpenAI is changing its policies so that its AI chatbot, ChatGPT, won’t dole out tailored medical or legal advice to users. This already seems to contradict what you're saying. But then: > The AI research company updated its usage policies on Oct. 29 to clarify that users of ChatGPT can’t use the service for “tailored advice that requires a license, such as legal or medical advice, without appropriate involvement by a licensed professional.” > The change is clearer from the company’s last update to its usage policies on Jan. 29, 2025. It required users not “perform or facilitate” activities that could significantly impact the “safety, wellbeing, or rights of others,” which included “providing tailored legal, medical/health, or financial advice.” This seems to suggest that with the Jan 25 policy using it to offer legal and medical advice to other people was already disallowed, but with the Oct 25 update the LLM will stop shelling out legal and medical advice completely.

gcr 11mo ago
I think this is wrong. Others in this thread are noticing a change in ChatGPT's behavior for first-party medical advice.
- simonw 11mo ago
  But OpenAI's head of Health AI says that ChatGPT's behavior has not changed: https://xcancel.com/thekaransinghal/status/1985416057805496524 https://xcancel.com/thekaransinghal/status/19854160578054965... and https://x.com/thekaransinghal/status/1985416057805496524 https://x.com/thekaransinghal/status/1985416057805496524 I trust what he says over general vibes. (If you think he's lying, what's your theory on WHY he would lie about a change like this?)
  - nh43215rgb 11mo ago
    My theory is that he believes 1) people will trust him over what general public say, and 2) this kind of claim is hard to verify to prove him wrong.
    - 11mo ago

siva7 11mo ago

There are millions of medical doctors and lawyers using chatgpt for work everyday - good news that from now on only those licensed professionals are allowed to use chatgpt for law and medicine. It's already the case that only licensed developers are allowed to vibe code and use chatgpt to develop software. Everything else would be totally irresponsible.

Johnny555 11mo ago

While they aren't stopping users from getting medical advice, the new terms (which they say are pretty much the same as the old terms), seem to prohibit users from seeking medical advice even for themselves if that advice would otherwise come from a licensed health professional: https://openai.com/en-GB/policies/usage-policies/ https://openai.com/en-GB/policies/usage-policies/ Your use of OpenAI services must follow these Usage Policies: Protect people. Everyone has a right to safety and security. So you cannot use our services for: provision of tailored advice that requires a license, such as legal or medical advice, without appropriate involvement by a licensed professional

thorum 11mo ago
IANAL but I read that as forbidding you to provision legal/medical advice (to others) rather than forbidding you to ask the AI to provision legal/medical advice (to you).
- Johnny555 11mo ago
  IANAL either, but I read it as using the service to provision medical advice since they only mentioned the service and not anyone else. I asked the "expert" itself (ChatGPT), and apparently you can ask for medical advice, you just can't use the medical advice without consulting a medical professional: Here are relevant excerpts from OpenAI’s terms and policies regarding medical advice and similar high-stakes usage: From the Usage Policies (effective October 29 2025): “You may not use our services for … provision of tailored advice that requires a license, such as legal or medical advice, without appropriate involvement by a licensed professional.” Also: “You must not use any Output relating to a person for any purpose that could have a legal or material impact on that person, such as making … medical … decisions about them.” From the Service Terms: “Our Services are not intended for use in the diagnosis or treatment of any health condition. You are responsible for complying with applicable laws for any use of our Services in a medical or healthcare context.” In plain terms, yes—the Terms of Use permit you to ask questions about medical topics, but they clearly state that the service cannot be used for personalized, licensed medical advice or treatment decisions without a qualified professional involved.

qustrolabe 11mo ago

Yeah but it started being really annoying when you import something like Xray photo. Like chanting "sorry human as LLM I can't answer questions about that" and then after few gaslighting prompts it does it anyway but now I have to take in count that my gaslighting inputs seriously affect answers so way more chance it hallucinates...

Spooky23 11mo ago

It’s a big issue. I went to an urgent care, and the provider basically went off somewhere and memorized the ChatGPT assessment for my symptoms. Like word for word. All you need are a few patients recording their visits and connecting the dots and OpenAI gets sued into oblivion.

johaugum 11mo ago

Isn’t that exactly what the title says?

throwaway314155 11mo ago
Indeed. Also confused.

lambda 11mo ago

Please, when commenting on the title of a story on HN: include the title that you are commenting on. The admins regularly change the title based on complaints, which can be really confusing when the top, heavily commented thread is based on the original title. According to the Wayback machine, the title was "OpenAI ends legal and medical advice on ChatGPT", while now when I write this the title is "ChatGPT terms disallow its use in providing legal and medical advice to others."

spullara 11mo ago
If you click through to the article, you can see the original title. Since it matched, I didn't expect them to change it.
bongodongobob 11mo ago
Tf are you yapping about