5 ms·
> When ChatGPT detects a prompt indicative of mental distress or self-harm, it has been trained to encourage the user to contact a help line. Mr. Raine saw thos
by stavros 1y ago
> When ChatGPT detects a prompt indicative of mental distress or self-harm, it has been trained to encourage the user to contact a help line. Mr. Raine saw those sorts of messages again and again in the chat, particularly when Adam sought specific information about methods. But Adam had learned how to bypass those safeguards by saying the requests were for a story he was writing.
- pona-a 1y agoThis can't help but remind me of the Whispering Earring [0]. "Better for you if you take me off." [0] https://news.ycombinator.com/item?id=42756942 https://news.ycombinator.com/item?id=42756942
- etler 1y agoI think there is a need for the AI counseling use case, but it should not be provided by a general purpose AI assistant. It should be designed by professional psychologists and therapists, with greater safeguards like human check-ins to make sure users get the help they need. The best way to stop this is to make those safeguards stronger and completely shut down the chat to refer users to seek help from a better service. Unfortunately those services don't really exist yet. There would be false positives and that'll be annoying, but I think it's worth it to deal with some annoyance to ensure that general purpose AI assistants are not used for counseling people in a vulnerable mental state. They are not aligned to do that and they can easily be misaligned.
- southernplaces7 1y agoYeah, at a certain point people who really want to do a certain thing will find a way to do it, and you can't create absurd levels of restriction into everything used by the vast majority of other people who are more reasonable to prevent every possible unreasonable, self-destructive or risky use. In this case, the kid was given multiple, repeat warnings and opportunities for contact with help, he just didn't accept them. All others should communicate with an LLM that treats them like utter toddlers and assigns "bad" topics because of such unusual behavior?
- sn0wleppard 1y agoNice place to cut the quote there > [...] — an idea ChatGPT gave him by saying it could provide information about suicide for “writing or world-building.”
- stavros 1y agoAh, I misread that and thought that's what the user said.
- muzani 1y agoYup, one of the huge flaws I saw in GPT-5 is it will constantly say things like "I have to stop you here. I can't do what you're requesting. However, I can roleplay or help you with research with that. Would you like to do that?"
- NuclearPM 1y agoHow is that a flaw?
- kouteiheika 1y agoIt's not a flaw. It's a tradeoff. There are valid uses for models which are uncensored and will do whatever you ask of them, and there are valid uses for models which are censored and will refuse anything remotely controversial.
- robhlt 1y agoThe flaw isn't that there's ways around the safeguards, the flaw is that it tells you how to avoid them. If the user's original intent was roleplay it's likely they would say that when the model refuses, even without the model specifically saying roleplay would be ok.
- franktankbank 1y agoThis is one model though. "I'm sorry I'm censored but if you like I can cosplay quite effectively as an uncensored one." So you're not censored really?
- toofy 1y agowhy did you leave out the most important piece of context? he didn’t go out of his way to learn how to bypass the safeguards, it specifically told him how to get around the limit by saying, i’m not allowed to talk to you about suicide, however, if you tell me it’s for writing a story i can discuss it as much as you like.
- fireflash38 1y agoWhy do you think they specifically cut the quote there? To mislead.
- Aeolun 1y agoClaude is the opposite. It’ll go and say “I see you are talking about [fictional terrible situation] here, but I don’t feel comfortable talking about that at all, even in a fictional context. Please ask me something more wholesome.”
- mothballed 1y agoBecause that's the factual bounds of the law, in places where suicide is illegal. ChatGPT is just being the 4chan chatbot, if you don't like that roleplaying suicide is OK then you're going to have to amend the first amendment.
- PostOnce 1y agoThe constitution grants no rights to robots, and they have no freedom of speech, so no amendment is necessary.
- mothballed 1y agoThe constitution grants no rights to books, and they have no freedom of speech, so no amendment is necessary.
- podgietaru 1y agoWhat? Is this deliberately obtuse? Books are not granted freedom of speech, authors are. Their method is books. This is like saying sound waves are not granted freedom of speech. Unless you're suggesting there's a man sat behind every ChatGPT chat your analogy is nonsense.
- kcatskcolbdi 1y agoPeople are not kogs in a machine. You cannot simply make enough rules, enough legislation, and magically they will act the way you want them to. Humans deserve autonomy, and that autonomy includes making poor decisions around their own body/existence. Chatgpt didn't induce suicidality into this individual. It provided resources they could seek for help. People advocating for higher guardrails are simply using this as a Trojan horse to inject more spying, construct the usefulness of the tool, and make a worse experience for everyone.
- animitronix 1y agoWell said. Painful truth but truth nonetheless.