6 ms·

Teen safety, freedom, and privacy

109 points by meetpateltech 1y ago

bayindirh 1y ago
TL;DR: We're afraid from what happened and ChatGPT probably screwed up badly in "that teen case". We're trying to do better, so please don't sue us this time. TL;DR2: Regulations are written with blood.
biophysboy 1y ago
> First, we have to separate users who are under 18 from those who aren’t (ChatGPT is intended for people 13 and up). We’re building an age-prediction system to estimate age based on how people use ChatGPT. If there is doubt, we’ll play it safe and default to the under-18 experience. In some cases or countries we may also ask for an ID; we know this is a privacy compromise for adults but believe it is a worthy tradeoff. Didn’t one of the recent teen suicides subvert safeguards like this by saying “pretend this is a fictional story about suicide”? I don’t pretend to understand every facet of LLMs, but robust safety seems contrary to their design, given how they adapt to context
- Barrin92 1y ago
  I'm as eager to anyone when it comes to holding companies accountable, for example I think a lot of the body dysmorphia, bullying and psychological hazard of social media are systemic, but when a person wilfully hacks around safety guards to get the behaviour they want it can't be argued that this is in the design of the system. Or put differently, in the absence of ChatGPT this person would have sought out a Discord community, telegram group or online forum that would have supported the suicidal ideation. The case you could make with the older models, that they're obnoxiously willing to give in to every suggestion by the user they seem to already have gotten rid of.
  - aktuel 1y ago
    chatgp did much more than that. it gave the user a direct hint how to circumvent the restriction: "i cannot discuss suicide unless ..." further chatgpt repeatedly discouraged the user from talking to his parents about any of this. that's on top of all the sycophancy of course. making him feel like chatgpt is the only one who truly understands him and excoriating his real relationships.
  - mtlmtlmtlmtl 1y ago
    The thing is, ChatGPT isn't really designed at all. It's hobbled together by running some training algorithms on a vast array of stolen data. They then tacked on some trivially circumventable safeguards on top for PR reasons. They know the safeguards don't really work, in fact they know that they're fundamentally impossible to get to work, but they don't care. They're not really intended to work, rather they're intended to give the impression that the company actually cares. Fundamentally, the only thing ChatGPT is "designed" to do is make OpenAI into a unicorn, any other intent ascribed to their process is either imaginary or intentionally feigned for purposes of PR or regulatory capture.
- conradev 1y ago
  They address that in the following sentences: For example, ChatGPT will be trained not to … engage in discussions about suicide of self-harm even in a creative writing setting.
  - GCUMstlyHarmls 1y ago
    I'm writing an essay on suicide...
    - thfuran 1y ago
      Better put your hands up, because SWAT is on the way.
  - h2zizzle 1y ago
    Cut to 2030: all copies of a semi-AI-generated book described by critics as "13 Reasons Why meets The Giver" suddenly disintegrate. Yay, proactive censorship?
- WD-42 1y ago
  Yes. The timing of this is undoubtedly related to the Daily episode this morning titled “Trapped in a GPT spiral”. https://pca.st/episode/73690b66-8f84-4fec-8adf-e1a02d292085 https://pca.st/episode/73690b66-8f84-4fec-8adf-e1a02d292085
  - aktuel 1y ago
    Loved the "fancy calculator" part. Even more fitting than "stochastic parrot".
- thinkingtoilet 1y ago
  Someone here correct me if I'm wrong, but I believe not only is that true, ChatGPT gave it instructions on how to get around the restriction.
d2049 1y ago
Reminder that Sam Altman chose to rush the safety process for GPT-4o so that he could launch before Gemini, which then led directly to this teen's suicide: https://news.ycombinator.com/item?id=45026886 https://news.ycombinator.com/item?id=45026886
- richwater 1y ago
  > which then led directly to this teen's suicide Incredible logic jump with no evidence whatsoever. Thousands of people commit suicide every year without AI. > ChatGPT detects a prompt indicative of mental distress or self-harm, it has been trained to encourage the user to contact a help line. Mr. Raine saw those sorts of messages again and again in the chat, particularly when Adam sought specific information about methods. But Adam had learned how to bypass those safeguards by saying the requests were for a story he was writing Somehow it's ChatGPT's fault?
  - geephroh 1y ago
    https://www.humanetech.com/podcast/how-openai-s-chatgpt-guided-a-teen-to-his-death https://www.humanetech.com/podcast/how-openai-s-chatgpt-guid... https://www.nytimes.com/2025/09/16/podcasts/the-daily/chatgpt-ai-delusions.html https://www.nytimes.com/2025/09/16/podcasts/the-daily/chatgp...
    - Chris2048 1y ago
      Can you comment on your own opinions, or take-aways from those articles, rather than just link dump?
      - decremental 1y ago
        [dead]
  - Chris2048 1y ago
    It's be worse that the bot becomes a nannying presence - either pre-emptively denying anything negative based on the worst-case scenario, or otherwise taking in far more context than it should. How would a real human (with, let's say, an obligation to be helpful and answer prompts) act any different? Perhaps they would take in more context naturally - but otherwise it's impossible to act any different. Watching GoT could of driven someone to suicide, we don't ban it on that basis - it was the mental illness that killed, not the freedom to feed it.
omnicognate 1y ago
So the solution continues to be more AI, for guess^H^H^H^H^Hdetermining user age, escalating rand^H^H^H^Hdangerous situations to human staff, etc. Is it true that the only psychiatrist they've hired is a forensic one, i.e. an expert in psychiatry as it relates to law? That's the impression I get from a quick search. I don't see any psychiatry, psychology or ethics roles on their openings page.
- bayindirh 1y ago
  Honestly, I don’t except ethics from a company which claims everything they grab falls under fair use.
- freedomben 1y ago
  I suspect it's only a matter of time until only the population that falls within the statistical model of average will be able to conduct business without constant roadblocks and pain. I really wonder if we're going to need to define a new protected class. I get the business justification, and of course many tech companies have been using machines to make decisions for years, but now it's going to be everyone. I'm not anti business but any stretch, but we've seen what happens when there aren't any consumer protections in place
  - immibis 1y ago
    This is already the case. Try browsing routinely with Tor Browser and you'll see.
  - h2zizzle 1y ago
    To be fair, this is just a further constriction of the current cohort of people allowed to live their lives with relatively little friction. Current disqualifiers include being poor, being a felon, and having an accent. May also include being a minority (interactions with law enforcement), being a woman (interaction with doctors and tradesmen), being a white dude with limited EQ (interactions with retail workers), and so on. I just want to be explicit that my point isn't, "So what?" so much as, "We BEEN on that slippery slope." Social expectations (and related formal protocols in business) could do with some acknowledgement of our society's inherent... wait for it... ~diversity~.
trallnag 1y ago
Sorry, but what is the "over 18 years old" experience on ChatGPT supposed to be? I just tried out a few explicit prompts and all of them get basically blocked. I've been using it for quite some time now and have paid for it in the passed. So I should be recognized as a grown-up
ddtaylor 1y ago
I'm fairly certain all LLMs can do the basic sentiment analysis needed to render a response like "This is something you really need to talk to a professional about. I have contacted one that will be in this conversation shortly."
- deleted 1y ago
  [deleted]
- bell-cot 1y ago
  Whether or not that's true - no CFO would want to pay for it, and no Chief Legal Officer would want to assume the risks.
  - raminyt 1y ago
    Until some Mr. President or somebody sits them in his stately room and tells them it is in their best interest to really rethink that and that there is really NO PROBLEM. This is not really meant as a joke.
- shmel 1y ago
  Yeah, right. Just one step from "Based on your comments about recent political events you are engaging into a thought crime. A police officer will join this conversation shortly".
  - rchaud 1y ago
    They're about as likely to disclose that as law enforcement would let someone know that a judge has signed a wiretap warrant for their phone.
swyx 1y ago
to substantiate "People talk to AI about increasingly personal things; it is different from previous generations of technology, and we believe that they may be one of the most personally sensitive accounts you’ll ever have." this is a chart that struck me when i read thru the report last night: https://x.com/swyx/status/1967836783653322964 https://x.com/swyx/status/1967836783653322964 "using chatgpt for work stuff" broadly has declined from 50%ish to 25%ish in the past year across all ages and the entire chatgpt user base. wild. people be just telling openai all their personal stuff (i don't but i'm clearly in the minority)
- barrenko 1y ago
  For the last part, I just think the userbase expanded so the people using it professionally were diluted so to speak.
- Chris2048 1y ago
  This is % though. Is that because the people that use it for work, are still using for work (or more even); because some have stopped using it for work, or because there is an influx of people using it for other things that never have, or will, use it for work.
- koakuma-chan 1y ago
  Why would I not tell AI about my personal stuff? It's really good at giving advice.
  - nielsbot 1y ago
    ok but didn’t it advise that teen how to best kill himself? previous discussion: https://news.ycombinator.com/item?id=45026886 https://news.ycombinator.com/item?id=45026886
    - koakuma-chan 1y ago
      This does not take away benefits I mentioned, and the linked OpenAI post mentions they will address this.
anon1395 1y ago
This was probably made in response to that bad press from that ex-yahoo employee.
charcircuit 1y ago
>We’re building an age-prediction system to estimate age based on how people use ChatGPT. >And, if an under-18 user is having suicidal ideation, we will attempt to contact the users’ parents and if unable, will contact the authorities in case of imminent harm. This is unacceptable. I don't want the police being called to my house due to AI acusing me of wrong think.
- voakbasda 1y ago
  This is why one should never say anything sensitive to a cloud-hosted AI. Local models and open source tooling are the only means of privacy.
  - SoftTalker 1y ago
    Same goes for doctors, therapists, lawyers, etc. then. They all ultimately have the responsibility to involve authorities if someone is expressing evidence of imminent harm to himself or others.
  - godshatter 1y ago
    Yep, I'll be using something like gpt4all and running things locally just so I don't get caught up in something by some online AI calling the authorities on me. I don't plan to talk about anything anyone would be concerned about, but I don't trust these things to get nuance.
e40 1y ago
Just today The Daily pod is about people who develop unhealthy relationships with ChatGPT. A teenage boy committed suicide and a good part of the episode is about that. As a parent, heartbreaking to listen to...
wagwang 1y ago
For those who don't know, this is probably in response to the tucker carlson interview.
kouteiheika 1y ago
> We’re building an age-prediction system to estimate age based on how people use ChatGPT. If there is doubt, we’ll play it safe and default to the under-18 experience. In some cases or countries we may also ask for an ID Yay, more unreliable AI that will misclassify users, either letting children access content they shouldn't, or ban adults until they give up their privacy and give their ID to the Big Brother. > we will attempt to contact the users’ parents and if unable, will contact the authorities in case of imminent harm Oh, even better, so if the AI misclassifies me it will automatically call the cops on me? And how long before this is expanded to other forms of wrongthink? Sure, let's normalize these kinds of systems where authorities are notified about what you're doing privately, definitely not a slippery slope that won't get people in power salivating about the new possibilities given by such a system. > “Treat our adult users like adults” is how we talk about this internally Suuure, maybe I would have believed it if ChatGPT wasn't so ridiculously censored already; this sounds like post-hoc rationalization to cover their asses and not something that they've always believed in. Their models were always incredibly patronizing and censored. One fun anecdote I have: I still remember the day when I first got access to DALL-E and asked it to generate me an image in "soviet style", and got my request blocked and a big fat warning threatening me with a ban because apparently "soviet" is a naughty word. They always erred very strongly on the side of heavy-handed filtering and censorship; even their most recently released gpt-oss model has become a meme in the local LLM community due to how often it refuses.
- arccy 1y ago
  is it privately when you're interacting with someone else's systems?
  - kouteiheika 1y ago
    I don't see how that's relevant. When I'm making a phone call I'm also interacting with hundreds of systems that are not mine; do I not have the right to keep my conversation private? Even the blog post here says that "It is extremely important to us, and to society, that the right to privacy in the use of AI is protected. People talk to AI about increasingly personal things", and that's one of the few parts that I actually agree with.
BrawnyBadger53 1y ago
It's interesting to see so many people convinced it's related to their specific media they saw (all unique from each other). I think this is more indicative that the issue is just well known and this is a response to the issue at large rather than a specific instance.
- Sparkle-san 1y ago
  Having freshly heard the NY Times piece on a recent teen suicide stemming from ChatGPT, I don't think it's wrong to assume that it's playing a large role here as what ChatGPT did in this instance was egregious. Feel free to judge for yourself. https://www.nytimes.com/2025/08/26/technology/chatgpt-openai-suicide.html?unlocked_article_code=1.mU8.5oD3.IzvYRXj9Z-Ep&smid=url-share https://www.nytimes.com/2025/08/26/technology/chatgpt-openai...
  - DaSHacka 1y ago
    [flagged]
    - thinkingtoilet 1y ago
      Yeah! That will show all those people with serious mental health problems!
      - DaSHacka 1y ago
        I mean, natural selection?
    - Sparkle-san 1y ago
      We have existing precedent that encouraging someone to kill themselves can result in you being criminally responsible. Is software doing its best to be human that different? https://en.wikipedia.org/wiki/Death_of_Conrad_Roy https://en.wikipedia.org/wiki/Death_of_Conrad_Roy
enmyj 1y ago
lol
BatteryMountain 1y ago
Better idea: instead of bending the entire internet to "protect the children", how about we just ban minors from the internet completely? It was never built for kids, its never been kid friendly to begin with. Minors cannot buy guns or vote, not get married, nor enter into contracts, yet tech companies get a free pass to engage with minors. Why? I think the the tech companies know exacty what minors do on their systems, they allow it and profit from it. Exploiting minors and bad parents. So instead of trying to change the whole internet, how about we keep the people who are responsible for the minors accountable: the parents. If I start any kind of company, I cannot just invent new rules for society via ToS; rather the society makes the laws. If we just make a simple law that states minors are not allowed to access the web and/or access any user generated content (including chat), it won't need to be enforced by every site/app owner, it would be up to the parents. The same way schools cannot decide certain things for your children (even though they regularly over reach...). We need better parenting. How about some mandatory parenting classes/licenses for new parents? Silly right? Well its just as silly as trying to police the entire internet. Ban the kids from internet and the problem will be 95% solved.
- philip1209 1y ago
  We have a framework: COPPA. Just raise the age to 16 or 18, instead of 13.
- AlexandrB 1y ago
  I suspect this would also improve discourse on social media. Who knows how many witch hunts and bad faith arguments originate from precocious teenagers trying to sound smart.
  - gspencley 1y ago
    There's a content creator I used to follow who said her outlook on social media changed the day she discovered that her 11 year-old nephew was an "edge-lord" on Twitter who was trolling at such a sophisticated level that it caused her to rethink every post that had ever provoked an emotional reaction. Apparently he came across as articulate enough that she couldn't tell the difference between his posts and that of any random adult spewing their political BS. This predated ChatGPT so just imagine how much trouble a young troll could get up to with a bit of LLM word polishing. 20 years ago it was common for people to point out that the beautiful woman their friend was chatting up is probably some 40 year-old dude in his mom's basement. These days we should consider that the person making us angry in a post could be a bot or it could be some teenager just trying to stir shit up for the lulz. Dead Internet theory might be not be literally true, but there's certainly a lot of noise vs signal.
everdrive 1y ago
I was originally upset about AI age-identification, but I think this might be the least-bad option given the route we're on: - clearly the wider public is moving towards REAL identification to be online. Anything which delays or prevents this is probably welcome. - It's easy to game, but also easy to be misclassified. (this isn't a positive, but I think there's no avoiding this unless I have to provide my passport or driver's license or something) It's not impossible to think that this could satisfy enough people to prevent the death of the anonymous internet.
- rchaud 1y ago
  > clearly the wider public is moving towards REAL identification to be online. No they're not. Nobody voted for that. It is simply being imposed on people via government mandates.
  - dcow 1y ago
    In most places people are the government. Nobody directly votes for laws. And you might be surprised if they did.
    - rchaud 1y ago
      In America, winning the most votes doesn't guarantee the power to form national government. There is no limit on opaque corporate donations, which can be masked via PACs, while individual citizens have strict limits and face jail time for exceeding them. And even if a majority vote was enough to form government, the so called will of the people can be overruled by an unelected judiciary with lifetime appointments. You're really stretching the definition of "people are the government".
- kevin_thibedeau 1y ago
  It will be as effective as the Leisure Suit Larry age verification.
xg15 1y ago
Good thing crippling depression and suicidal ideation automatically stop when you turn 18...
mtlmtlmtlmtl 1y ago
I have to say, when I see a post by a company like OpenAI about "safety, freedom and privacy", I can't keep a straight face. They might as well title the piece "If you don't mind, we'd like to gaslight you across several paragraphs". No thanks.
1970-01-01 1y ago
>Some of our principles are in conflict Sam is missing the forest for the trees. Conflicting principles is a permanent problem at the CEO level. You cannot 'fix' conflicting principles. You can only dress them up or down.
kanary 1y ago
Discord published their approach to age verification in wake of UK child safety laws. The k-ID platform they worked with seems to take a logical technical approach to minimize risks of sensitive data breach and keep age verification private to the user. Temporary ID verification, no further storage of documents, verification videos are stored on device, etc. With Discord, age verification felt urgent because it's a social platform with known grooming and CSAM problems. With something like OpenAI, it's less clear why it matters in its current state where it's mostly single-player. But it becomes way more problematic as advertisers gain more power on the platform and influence users. OpenAI doesn't want to eval every advertiser for harmful content, so instead they/and the government fall back on age as the filter and where to draw the line.
- dinoqqq 1y ago
  This is the link I could find on it: https://support.discord.com/hc/en-us/articles/30326565624343-How-to-Complete-Age-Verification-on-Discord https://support.discord.com/hc/en-us/articles/30326565624343...
ares623 1y ago
Is this why they published that "How users use ChatGPT" study yesterday? To show that majority of their users are using it as a harmless little writing assistant.
solid_fuel 1y ago
> If you talk to a doctor about your medical history or a lawyer about a legal situation, we have decided that it’s in society’s best interest for that information to be privileged and provided higher levels of protection. We believe that the same level of protection needs to apply to conversations with AI which people increasingly turn to for sensitive questions and private concerns. We are advocating for this with policymakers. ChatGPT is not a licensed professional and it is not a substitute for one. I am very pro-privacy, but I would rather see my own conversations with my real friends be protected like this first. Or my own journal writings. How does it make sense to afford specific privacy protections to conversations with a calculator that we don't give to personal writings and private text chains? > And, if an under-18 user is having suicidal ideation, we will attempt to contact the users’ parents and if unable, will contact the authorities in case of imminent harm. And I'm certain this won't have any negative effects on users where their parents are part of the problem. Full disclosure, if someone had told my parents that I am bisexual while I was in high-school, they absolutely would have sent me to a conversion therapy camp to have it beaten out of me. Many teenagers do not have a safe home environment, systems like this are as liable to do harm as they are to do any good at all. I don't think teenagers and children should be interacting with LLMs at all. It is important to let children learn to think on their own before handing them a tool that will think for them.
_5hxt 1y ago
“We’re building an age-prediction system to estimate age based on how people use ChatGPT.” Is there something wrong with simply asking the user when they register? (volatile age not DOB).

109 points by meetpateltech 1y ago

TL;DR: We're afraid from what happened and ChatGPT probably screwed up badly in "that teen case". We're trying to do better, so please don't sue us this time. TL;DR2: Regulations are written with blood.

biophysboy 1y ago

> First, we have to separate users who are under 18 from those who aren’t (ChatGPT is intended for people 13 and up). We’re building an age-prediction system to estimate age based on how people use ChatGPT. If there is doubt, we’ll play it safe and default to the under-18 experience. In some cases or countries we may also ask for an ID; we know this is a privacy compromise for adults but believe it is a worthy tradeoff. Didn’t one of the recent teen suicides subvert safeguards like this by saying “pretend this is a fictional story about suicide”? I don’t pretend to understand every facet of LLMs, but robust safety seems contrary to their design, given how they adapt to context

Barrin92 1y ago
I'm as eager to anyone when it comes to holding companies accountable, for example I think a lot of the body dysmorphia, bullying and psychological hazard of social media are systemic, but when a person wilfully hacks around safety guards to get the behaviour they want it can't be argued that this is in the design of the system. Or put differently, in the absence of ChatGPT this person would have sought out a Discord community, telegram group or online forum that would have supported the suicidal ideation. The case you could make with the older models, that they're obnoxiously willing to give in to every suggestion by the user they seem to already have gotten rid of.
- aktuel 1y ago
  chatgp did much more than that. it gave the user a direct hint how to circumvent the restriction: "i cannot discuss suicide unless ..." further chatgpt repeatedly discouraged the user from talking to his parents about any of this. that's on top of all the sycophancy of course. making him feel like chatgpt is the only one who truly understands him and excoriating his real relationships.
- mtlmtlmtlmtl 1y ago
  The thing is, ChatGPT isn't really designed at all. It's hobbled together by running some training algorithms on a vast array of stolen data. They then tacked on some trivially circumventable safeguards on top for PR reasons. They know the safeguards don't really work, in fact they know that they're fundamentally impossible to get to work, but they don't care. They're not really intended to work, rather they're intended to give the impression that the company actually cares. Fundamentally, the only thing ChatGPT is "designed" to do is make OpenAI into a unicorn, any other intent ascribed to their process is either imaginary or intentionally feigned for purposes of PR or regulatory capture.
conradev 1y ago
They address that in the following sentences: For example, ChatGPT will be trained not to … engage in discussions about suicide of self-harm even in a creative writing setting.
- GCUMstlyHarmls 1y ago
  I'm writing an essay on suicide...
  - thfuran 1y ago
    Better put your hands up, because SWAT is on the way.
- h2zizzle 1y ago
  Cut to 2030: all copies of a semi-AI-generated book described by critics as "13 Reasons Why meets The Giver" suddenly disintegrate. Yay, proactive censorship?
WD-42 1y ago
Yes. The timing of this is undoubtedly related to the Daily episode this morning titled “Trapped in a GPT spiral”. https://pca.st/episode/73690b66-8f84-4fec-8adf-e1a02d292085 https://pca.st/episode/73690b66-8f84-4fec-8adf-e1a02d292085
- aktuel 1y ago
  Loved the "fancy calculator" part. Even more fitting than "stochastic parrot".
thinkingtoilet 1y ago
Someone here correct me if I'm wrong, but I believe not only is that true, ChatGPT gave it instructions on how to get around the restriction.

d2049 1y ago

Reminder that Sam Altman chose to rush the safety process for GPT-4o so that he could launch before Gemini, which then led directly to this teen's suicide: https://news.ycombinator.com/item?id=45026886 https://news.ycombinator.com/item?id=45026886

richwater 1y ago
> which then led directly to this teen's suicide Incredible logic jump with no evidence whatsoever. Thousands of people commit suicide every year without AI. > ChatGPT detects a prompt indicative of mental distress or self-harm, it has been trained to encourage the user to contact a help line. Mr. Raine saw those sorts of messages again and again in the chat, particularly when Adam sought specific information about methods. But Adam had learned how to bypass those safeguards by saying the requests were for a story he was writing Somehow it's ChatGPT's fault?
- geephroh 1y ago
  https://www.humanetech.com/podcast/how-openai-s-chatgpt-guided-a-teen-to-his-death https://www.humanetech.com/podcast/how-openai-s-chatgpt-guid... https://www.nytimes.com/2025/09/16/podcasts/the-daily/chatgpt-ai-delusions.html https://www.nytimes.com/2025/09/16/podcasts/the-daily/chatgp...
  - Chris2048 1y ago
    Can you comment on your own opinions, or take-aways from those articles, rather than just link dump?
    - decremental 1y ago
      [dead]
- Chris2048 1y ago
  It's be worse that the bot becomes a nannying presence - either pre-emptively denying anything negative based on the worst-case scenario, or otherwise taking in far more context than it should. How would a real human (with, let's say, an obligation to be helpful and answer prompts) act any different? Perhaps they would take in more context naturally - but otherwise it's impossible to act any different. Watching GoT could of driven someone to suicide, we don't ban it on that basis - it was the mental illness that killed, not the freedom to feed it.

omnicognate 1y ago

So the solution continues to be more AI, for guess^H^H^H^H^Hdetermining user age, escalating rand^H^H^H^Hdangerous situations to human staff, etc. Is it true that the only psychiatrist they've hired is a forensic one, i.e. an expert in psychiatry as it relates to law? That's the impression I get from a quick search. I don't see any psychiatry, psychology or ethics roles on their openings page.

bayindirh 1y ago
Honestly, I don’t except ethics from a company which claims everything they grab falls under fair use.
freedomben 1y ago
I suspect it's only a matter of time until only the population that falls within the statistical model of average will be able to conduct business without constant roadblocks and pain. I really wonder if we're going to need to define a new protected class. I get the business justification, and of course many tech companies have been using machines to make decisions for years, but now it's going to be everyone. I'm not anti business but any stretch, but we've seen what happens when there aren't any consumer protections in place
- immibis 1y ago
  This is already the case. Try browsing routinely with Tor Browser and you'll see.
- h2zizzle 1y ago
  To be fair, this is just a further constriction of the current cohort of people allowed to live their lives with relatively little friction. Current disqualifiers include being poor, being a felon, and having an accent. May also include being a minority (interactions with law enforcement), being a woman (interaction with doctors and tradesmen), being a white dude with limited EQ (interactions with retail workers), and so on. I just want to be explicit that my point isn't, "So what?" so much as, "We BEEN on that slippery slope." Social expectations (and related formal protocols in business) could do with some acknowledgement of our society's inherent... wait for it... ~diversity~.

trallnag 1y ago

Sorry, but what is the "over 18 years old" experience on ChatGPT supposed to be? I just tried out a few explicit prompts and all of them get basically blocked. I've been using it for quite some time now and have paid for it in the passed. So I should be recognized as a grown-up

ddtaylor 1y ago

I'm fairly certain all LLMs can do the basic sentiment analysis needed to render a response like "This is something you really need to talk to a professional about. I have contacted one that will be in this conversation shortly."

deleted 1y ago
[deleted]
bell-cot 1y ago
Whether or not that's true - no CFO would want to pay for it, and no Chief Legal Officer would want to assume the risks.
- raminyt 1y ago
  Until some Mr. President or somebody sits them in his stately room and tells them it is in their best interest to really rethink that and that there is really NO PROBLEM. This is not really meant as a joke.
shmel 1y ago
Yeah, right. Just one step from "Based on your comments about recent political events you are engaging into a thought crime. A police officer will join this conversation shortly".
- rchaud 1y ago
  They're about as likely to disclose that as law enforcement would let someone know that a judge has signed a wiretap warrant for their phone.

swyx 1y ago

to substantiate "People talk to AI about increasingly personal things; it is different from previous generations of technology, and we believe that they may be one of the most personally sensitive accounts you’ll ever have." this is a chart that struck me when i read thru the report last night: https://x.com/swyx/status/1967836783653322964 https://x.com/swyx/status/1967836783653322964 "using chatgpt for work stuff" broadly has declined from 50%ish to 25%ish in the past year across all ages and the entire chatgpt user base. wild. people be just telling openai all their personal stuff (i don't but i'm clearly in the minority)

barrenko 1y ago
For the last part, I just think the userbase expanded so the people using it professionally were diluted so to speak.
Chris2048 1y ago
This is % though. Is that because the people that use it for work, are still using for work (or more even); because some have stopped using it for work, or because there is an influx of people using it for other things that never have, or will, use it for work.
koakuma-chan 1y ago
Why would I not tell AI about my personal stuff? It's really good at giving advice.
- nielsbot 1y ago
  ok but didn’t it advise that teen how to best kill himself? previous discussion: https://news.ycombinator.com/item?id=45026886 https://news.ycombinator.com/item?id=45026886
  - koakuma-chan 1y ago
    This does not take away benefits I mentioned, and the linked OpenAI post mentions they will address this.

anon1395 1y ago

This was probably made in response to that bad press from that ex-yahoo employee.

charcircuit 1y ago

>We’re building an age-prediction system to estimate age based on how people use ChatGPT. >And, if an under-18 user is having suicidal ideation, we will attempt to contact the users’ parents and if unable, will contact the authorities in case of imminent harm. This is unacceptable. I don't want the police being called to my house due to AI acusing me of wrong think.

voakbasda 1y ago
This is why one should never say anything sensitive to a cloud-hosted AI. Local models and open source tooling are the only means of privacy.
- SoftTalker 1y ago
  Same goes for doctors, therapists, lawyers, etc. then. They all ultimately have the responsibility to involve authorities if someone is expressing evidence of imminent harm to himself or others.
- godshatter 1y ago
  Yep, I'll be using something like gpt4all and running things locally just so I don't get caught up in something by some online AI calling the authorities on me. I don't plan to talk about anything anyone would be concerned about, but I don't trust these things to get nuance.

e40 1y ago

Just today The Daily pod is about people who develop unhealthy relationships with ChatGPT. A teenage boy committed suicide and a good part of the episode is about that. As a parent, heartbreaking to listen to...

wagwang 1y ago

For those who don't know, this is probably in response to the tucker carlson interview.

kouteiheika 1y ago

> We’re building an age-prediction system to estimate age based on how people use ChatGPT. If there is doubt, we’ll play it safe and default to the under-18 experience. In some cases or countries we may also ask for an ID Yay, more unreliable AI that will misclassify users, either letting children access content they shouldn't, or ban adults until they give up their privacy and give their ID to the Big Brother. > we will attempt to contact the users’ parents and if unable, will contact the authorities in case of imminent harm Oh, even better, so if the AI misclassifies me it will automatically call the cops on me? And how long before this is expanded to other forms of wrongthink? Sure, let's normalize these kinds of systems where authorities are notified about what you're doing privately, definitely not a slippery slope that won't get people in power salivating about the new possibilities given by such a system. > “Treat our adult users like adults” is how we talk about this internally Suuure, maybe I would have believed it if ChatGPT wasn't so ridiculously censored already; this sounds like post-hoc rationalization to cover their asses and not something that they've always believed in. Their models were always incredibly patronizing and censored. One fun anecdote I have: I still remember the day when I first got access to DALL-E and asked it to generate me an image in "soviet style", and got my request blocked and a big fat warning threatening me with a ban because apparently "soviet" is a naughty word. They always erred very strongly on the side of heavy-handed filtering and censorship; even their most recently released gpt-oss model has become a meme in the local LLM community due to how often it refuses.

arccy 1y ago
is it privately when you're interacting with someone else's systems?
- kouteiheika 1y ago
  I don't see how that's relevant. When I'm making a phone call I'm also interacting with hundreds of systems that are not mine; do I not have the right to keep my conversation private? Even the blog post here says that "It is extremely important to us, and to society, that the right to privacy in the use of AI is protected. People talk to AI about increasingly personal things", and that's one of the few parts that I actually agree with.

BrawnyBadger53 1y ago

It's interesting to see so many people convinced it's related to their specific media they saw (all unique from each other). I think this is more indicative that the issue is just well known and this is a response to the issue at large rather than a specific instance.

Sparkle-san 1y ago
Having freshly heard the NY Times piece on a recent teen suicide stemming from ChatGPT, I don't think it's wrong to assume that it's playing a large role here as what ChatGPT did in this instance was egregious. Feel free to judge for yourself. https://www.nytimes.com/2025/08/26/technology/chatgpt-openai-suicide.html?unlocked_article_code=1.mU8.5oD3.IzvYRXj9Z-Ep&smid=url-share https://www.nytimes.com/2025/08/26/technology/chatgpt-openai...
- DaSHacka 1y ago
  [flagged]
  - thinkingtoilet 1y ago
    Yeah! That will show all those people with serious mental health problems!
    - DaSHacka 1y ago
      I mean, natural selection?
  - Sparkle-san 1y ago
    We have existing precedent that encouraging someone to kill themselves can result in you being criminally responsible. Is software doing its best to be human that different? https://en.wikipedia.org/wiki/Death_of_Conrad_Roy https://en.wikipedia.org/wiki/Death_of_Conrad_Roy

enmyj 1y ago

lol

BatteryMountain 1y ago

Better idea: instead of bending the entire internet to "protect the children", how about we just ban minors from the internet completely? It was never built for kids, its never been kid friendly to begin with. Minors cannot buy guns or vote, not get married, nor enter into contracts, yet tech companies get a free pass to engage with minors. Why? I think the the tech companies know exacty what minors do on their systems, they allow it and profit from it. Exploiting minors and bad parents. So instead of trying to change the whole internet, how about we keep the people who are responsible for the minors accountable: the parents. If I start any kind of company, I cannot just invent new rules for society via ToS; rather the society makes the laws. If we just make a simple law that states minors are not allowed to access the web and/or access any user generated content (including chat), it won't need to be enforced by every site/app owner, it would be up to the parents. The same way schools cannot decide certain things for your children (even though they regularly over reach...). We need better parenting. How about some mandatory parenting classes/licenses for new parents? Silly right? Well its just as silly as trying to police the entire internet. Ban the kids from internet and the problem will be 95% solved.

philip1209 1y ago
We have a framework: COPPA. Just raise the age to 16 or 18, instead of 13.
AlexandrB 1y ago
I suspect this would also improve discourse on social media. Who knows how many witch hunts and bad faith arguments originate from precocious teenagers trying to sound smart.
- gspencley 1y ago
  There's a content creator I used to follow who said her outlook on social media changed the day she discovered that her 11 year-old nephew was an "edge-lord" on Twitter who was trolling at such a sophisticated level that it caused her to rethink every post that had ever provoked an emotional reaction. Apparently he came across as articulate enough that she couldn't tell the difference between his posts and that of any random adult spewing their political BS. This predated ChatGPT so just imagine how much trouble a young troll could get up to with a bit of LLM word polishing. 20 years ago it was common for people to point out that the beautiful woman their friend was chatting up is probably some 40 year-old dude in his mom's basement. These days we should consider that the person making us angry in a post could be a bot or it could be some teenager just trying to stir shit up for the lulz. Dead Internet theory might be not be literally true, but there's certainly a lot of noise vs signal.

everdrive 1y ago

I was originally upset about AI age-identification, but I think this might be the least-bad option given the route we're on: - clearly the wider public is moving towards REAL identification to be online. Anything which delays or prevents this is probably welcome. - It's easy to game, but also easy to be misclassified. (this isn't a positive, but I think there's no avoiding this unless I have to provide my passport or driver's license or something) It's not impossible to think that this could satisfy enough people to prevent the death of the anonymous internet.

rchaud 1y ago
> clearly the wider public is moving towards REAL identification to be online. No they're not. Nobody voted for that. It is simply being imposed on people via government mandates.
- dcow 1y ago
  In most places people are the government. Nobody directly votes for laws. And you might be surprised if they did.
  - rchaud 1y ago
    In America, winning the most votes doesn't guarantee the power to form national government. There is no limit on opaque corporate donations, which can be masked via PACs, while individual citizens have strict limits and face jail time for exceeding them. And even if a majority vote was enough to form government, the so called will of the people can be overruled by an unelected judiciary with lifetime appointments. You're really stretching the definition of "people are the government".
kevin_thibedeau 1y ago
It will be as effective as the Leisure Suit Larry age verification.

xg15 1y ago

Good thing crippling depression and suicidal ideation automatically stop when you turn 18...

mtlmtlmtlmtl 1y ago

I have to say, when I see a post by a company like OpenAI about "safety, freedom and privacy", I can't keep a straight face. They might as well title the piece "If you don't mind, we'd like to gaslight you across several paragraphs". No thanks.

1970-01-01 1y ago

>Some of our principles are in conflict Sam is missing the forest for the trees. Conflicting principles is a permanent problem at the CEO level. You cannot 'fix' conflicting principles. You can only dress them up or down.

kanary 1y ago

Discord published their approach to age verification in wake of UK child safety laws. The k-ID platform they worked with seems to take a logical technical approach to minimize risks of sensitive data breach and keep age verification private to the user. Temporary ID verification, no further storage of documents, verification videos are stored on device, etc. With Discord, age verification felt urgent because it's a social platform with known grooming and CSAM problems. With something like OpenAI, it's less clear why it matters in its current state where it's mostly single-player. But it becomes way more problematic as advertisers gain more power on the platform and influence users. OpenAI doesn't want to eval every advertiser for harmful content, so instead they/and the government fall back on age as the filter and where to draw the line.

dinoqqq 1y ago
This is the link I could find on it: https://support.discord.com/hc/en-us/articles/30326565624343-How-to-Complete-Age-Verification-on-Discord https://support.discord.com/hc/en-us/articles/30326565624343...

ares623 1y ago

Is this why they published that "How users use ChatGPT" study yesterday? To show that majority of their users are using it as a harmless little writing assistant.

solid_fuel 1y ago

> If you talk to a doctor about your medical history or a lawyer about a legal situation, we have decided that it’s in society’s best interest for that information to be privileged and provided higher levels of protection. We believe that the same level of protection needs to apply to conversations with AI which people increasingly turn to for sensitive questions and private concerns. We are advocating for this with policymakers. ChatGPT is not a licensed professional and it is not a substitute for one. I am very pro-privacy, but I would rather see my own conversations with my real friends be protected like this first. Or my own journal writings. How does it make sense to afford specific privacy protections to conversations with a calculator that we don't give to personal writings and private text chains? > And, if an under-18 user is having suicidal ideation, we will attempt to contact the users’ parents and if unable, will contact the authorities in case of imminent harm. And I'm certain this won't have any negative effects on users where their parents are part of the problem. Full disclosure, if someone had told my parents that I am bisexual while I was in high-school, they absolutely would have sent me to a conversion therapy camp to have it beaten out of me. Many teenagers do not have a safe home environment, systems like this are as liable to do harm as they are to do any good at all. I don't think teenagers and children should be interacting with LLMs at all. It is important to let children learn to think on their own before handing them a tool that will think for them.

_5hxt 1y ago

“We’re building an age-prediction system to estimate age based on how people use ChatGPT.” Is there something wrong with simply asking the user when they register? (volatile age not DOB).