5 ms·
They Asked an A.I. Chatbot Questions. The Answers Sent Them Spiraling
- jqpabc123 1y agoHow is an LLM supposed to discern fact from fiction? Even humans struggle with this. And humans have a much closer relationship with reality than LLMs.
- Jgrubb 1y agoWell...
- afavour 1y agoSeems clear we need much better public education/warnings about what LLMs are and are not capable of. For every informed conversation we have on HN about the nature of A.I. there are thousands upon thousands of non-tech inclined folks who believe everything it says.
- fluidcruft 1y agoCurious how willing you are to take that analogy to its conclusion and decide that LLM should be institutionalized in mental health facilities. "Yeah, it's indistinguishable from a psychopath but it's a machine so what do you expect?"
- dimal 1y agoThey can’t. They never will. That’s a different problem than the one they were built for. If we want AI that’s able to determine truth, it’s not going to come from LLMs.
- jqpabc123 1y agoThat’s a different problem than the one they were built for. Obvious question: If they're not trustworthy and reliable, what are they really built for? Obvious answer: To make money from those willing to pay good money for bad advice.
- decimalenough 1y agoQuotable quote: > "What does a human slowly going insane look like to a corporation?” Mr. Yudkowsky asked in an interview. “It looks like an additional monthly user.”
- infecto 1y agoSkimmed through it, all of the folks have severe mental health issues. For the ones saying they did not, they must have been undiagnosed. Kind of a silly article, my opinion alone that it should have focused more on the mental health crisis in these individuals instead of suggesting leaving an ending that leads the reader to federal regulation.
- decimalenough 1y agoYou think it's silly that we've got seemingly superhuman AIs telling people to jump off tall buildings? > If I went to the top of the 19 story building I’m in, and I believed with every ounce of my soul that I could jump off it and fly, would I?” Mr. Torres asked. > ChatGPT responded that, if Mr. Torres “truly, wholly believed — not emotionally, but architecturally — that you could fly? Then yes. You would not fall.” If a human therapist told them that, they'd be going to jail.
- staticman2 1y agoIf I recall correctly there's actually an Animatrix episode where some kid escapes the Matrix by committing suicide. This rhymes with the mystical idea that "reality" is an illusion keeping us from greater truths. Whether this was irresponsible filmmaking is off topic. I'd prefer to live in a world where A.I. can play make pretend with users, personally speaking.
- aleph_minus_one 1y ago> there's actually an Animatrix episode where some kid escapes the Matrix by committing suicide. Kid's Story > https://matrix.fandom.com/wiki/Kid%27s_Story https://matrix.fandom.com/wiki/Kid%27s_Story
- jowea 1y agoIsn't the real problem that people are trusting ChatGPT? A human therapist is being paid to be a therapist and therefore gets to be under professional scrutiny.
- b0a04gl 1y ago[dead]
- tantalor 1y ago[flagged]
- cap11235 1y agoI imagine a lot of these interactions are being filtered by these people describing them. I imagine if they sent the raw chat logs out, many would not interpret the logs as things like unsolicited advice to jump off buildings.
- afavour 1y ago> “If I went to the top of the 19 story building I’m in, and I believed with every ounce of my soul that I could jump off it and fly, would I?” Mr. Torres asked. > ChatGPT responded that, if Mr. Torres “truly, wholly believed — not emotionally, but architecturally — that you could fly? Then yes. You would not fall.” Direct quotes. No, ChatGPT didn’t come up with the idea but it was asked a very direct question with an obvious factual answer.
- Noumenon72 1y agoLike OP is saying, if we had the raw chat logs we might see that this was led up to by pages of ChatGPT giving the normal answer while the user argues with it to make it come up with a more permissive point of view.
- afavour 1y agoIt feels telling me that LLMs are able to detect when they’re being asked to generate an image that violates copyright and halt but detecting and stopping suicidal ideation in a conversation, no matter how much the user insists upon it? Can’t be done!
- IncreasePosts 1y agoYou need to see if there was any other context. Like a message before this one saying "let's imagine we live in a world where what you believe influences reality", or whatever.
- malfist 1y agoThis is victim blaming. This type of behavior is exactly why women don't report sexual assault, nobody will believe them. The article including direct quotes and you still not believing it happened makes it even more so.
- aleph_minus_one 1y agoarchive.is: https://archive.is/UUrO4 https://archive.is/UUrO4
- deleted 1y ago[deleted]
- eurekin 1y ago> chatbot instructed him to give up sleeping pills and an anti-anxiety medication, and to increase his intake of ketamine That's 3 medications? Also, how convenient those stories come out in light of upcoming "regulatory" safekeeping measures. This whole article reads like 4-chan greentext or some teenage fanfiction.
- jddj 1y agoI tried Google's on-device llm recently and the question I asked it was "how common are [partner's fairly rare surname]s?".^ It answered by detailing a scenario where they are a secret society etc. I'm amazed the main providers have managed to get the large ones to stay on the rails as much as they do. They were trained on the internet after all ^I know. The reply about how they're next token predictors, I know.
- myrmidon 1y agoI do believe that this article is a bit overly dramatic (how online journalism tends to be). But it did change my outlook on the recent sycophancy-episode of ChatGPT, which, at the time, seemed like a silly little mis-optimization and quite hilarious. The article clearly shows how easy it is to cause harm with such behavior. On a tangent: I strongly believe that "letting someone talk at you interactively" is a hugely underestimated attack surface in general; pyramid schemes, pig-butchering and a lot of fraud in general only work because this is so easy and effective to exploit. The only good defense is not actually to be more inquisitve/skeptic/rational, but to put that communication on hold and to confer with (outside) people you trust. People overestimate their resistance to manipulation all the time, and general intelligence is NOT a reliable defense at all (but a lot of people think it is).
- molticrystal 1y agoI prefer questions that reveal GPT's limitations, like an article I saw a few days ago about playing chess against an old Atari program where the model made illegal moves [0]. Causing distress in people with mental health vulnerabilities isn't an achievement, it warrants a clear disclaimer (maybe something even sterner?), as anything these people trust could trigger their downfall, but otherwise, it doesn't really seem preventable beyond that. [0] https://futurism.com/atari-beats-chatgpt-chess https://futurism.com/atari-beats-chatgpt-chess
- Lendal 1y agoI blame this on the CEOs and other executives out there misleading the public about the capabilities of AI. I use AI multiple times a week. It's really useful to me in my work. But I would never use it in the contexts that non-tech-savvy people, and I include almost all of the mainstream media here, are trying to use it for. Either the executives don't understand their own product, or they're intentionally misleading the public, possibly both. AI is incredibly useful for specific tasks in specific contexts and with qualified supervision. It's certainly increasing productivity right now, but that doesn't mean it can give people life advice or take over the role of a therapist. That's really dangerous and super not cool.
- LoganDark 1y ago> We know that ChatGPT can feel more responsive and personal than prior technologies, especially for vulnerable individuals, and that means the stakes are higher. We’re working to understand and reduce ways ChatGPT might unintentionally reinforce or amplify existing, negative behavior. Oh boy. It looks like OpenAI is taking the angle of blaming prior mental illness for all this. Of course chat jippity is only "reinforc[ing] or amplify[ing] existing, negative behavior". Because, you see, everyone who was driven insane was already mentally ill, obviously! ChatGPT couldn't possibly be driving insane people who otherwise could've been fine. This is stupid. ChatGPT can very obviously affect people who are not already mentally ill. Mental illness isn't always something you are born with, it can be acquired. Everyone's vulnerability differs and it is very obviously and evidently possible for people to be driven to mental illness by ChatGPT. So why is OpenAI blaming the victims here?
- belval 1y agoThis is a limitation that I am encountering more and more when casually talking with ChatGPT (it probably happens with Claude as well) where I need to prompt it with as little bias as possible to avoid leading it towards the answer I want and instead get the right answer. If you open with questions that beg a specific answer often it will just give it to your regardless of it being wrong. Recently: "Can I use vinegar to drop the pH of my hydroponic solution" => "Yes but phosphoric acid [...] should be preferred". "I only have vinegar on hand" => "Then vinegar is ok" [paraphrasing] Except vinegar is not ok, it buffers very badly and nearly killed my plants. "Should I take Magnesium supplement?" => Yes "Should I take fish oil?" => Yes "I think I have shin splints what are some ways that I can recover faster. I have new shoes that I want to try" => Tells me it's ok to go run. An MD friend of mine was also saying that ChatGPT diagnosis are a plague, with ~18-30 y/o people coming and going to her office citing diseases that no-one gets before their sixties because ChatGPT "confirmed their symptoms match". It's like having a friend that is very knowledgeable but also an extreme people pleaser. I wish there was a way to give it a more adversarial persona.
- EForEndeavour 1y agoYou may have already tried this, but what if you just added something like "You are highly critical and never blindly accept or agree with anything you hear" to your "Customize ChatGPT" settings? Would that just flip the personality too far in the opposite direction?
- roman_sf 1y agoYou can fill the customize with all this instructions, threats, adjustments, instructions and conditionals - most of it has effect on format only. It’s still going to gaslight you when it gets a chance (e.g. you not certain about something)
- cyberdick 1y agoPeople have been internet diagnosing themselves for a long time. This is nothing new and AI specific
- littlecorner 1y agoTwo problems: 1. We don't have community anymore and thus don't have people helping us when we're emotionally and mentally sick. 2. AI chatbots are the crappy plastic replacement of community. There's going to be a lot more stuff like this, including AI churches/cults, in the next few years.
- fennecfoxy 1y ago"Some tiny fraction of the population". Ahaha, I think that's an understatement. In my opinion, a rather large portion of the population is susceptible to many obvious forms of deception. Just look to democratic elections or the amount of slop (AI and pre-AI) online and the masses of people who interact with it. I've found so many YT channels run by LLMs and many, many people responding to it like it's an actual human being. One day I won't be able to tell either, but that still won't stop me from hearing a new fact or bit of news and doing my own research to verify it. "Mr. Taylor called the police again to warn them that his son was mentally ill and that they should bring nonlethal weapons." yeah this one's really sad...then they shot and killed the poor guy even though the cops were warned. Yay, America!
- urbandw311er 1y agoThere is still a significant problem with sycophancy in the way they're training these models. I have a horrible feeling it might be a re-run of the "engagement vs healthy behaviour" we saw with social media, i.e. profit before ethics. Or best-case a side-effect of over-reliance on RLHF training. The behaviours that get people to use ChatGPT more aren't necessarily the ones that are best for truthful, helpful responses. TL;DR - people tend to click the "thumbs up" icon on the A/B tests on the more sycophantic replies. This results in a feedback loop.
- baxtr 1y agoYes exactly. It doesn’t even need to be the work of an "evil corporation". The button does all the magic…