9 ms·

Every AI would refuse the prompt. I was banned for researching Nordic assisted death and asking which drug exactly they administered (and what quantity). Claude

Every AI would refuse the prompt. I was banned for researching Nordic assisted death and asking which drug exactly they administered (and what quantity). Claude refused, alerted Anthropic, and I was banned a couple days later. Thankfully the appeal form worked, but by then I was using a different Claude account praying they didn’t ban me again. There’s an uncensored model floating around that you can run locally with llama.cpp: https://www.reddit.com/r/LocalLLaMA/comments/1rq7jtm/qwen3535ba3b_uncensored_aggressive_gguf_release/ https://www.reddit.com/r/LocalLLaMA/comments/1rq7jtm/qwen353... it’s annoying to use since you run out of context window quickly, and it’s certainly not able to be deployed in production (i.e. Tom Riddle’s diary as a service). For better or worse, fun is no longer allowed. It coincided with “AI psychosis” being coined as a term.

Terr_ 3mo ago
For a dash of dystopia: Imagine a company starts using that same LLM fuzzy-matching [0] against what you intend to be normal search activity, to detect "bad" queries and "bad" users who will be blocked. Maybe they'll delete your SSO/email/videos/photos too, who knows. I can easily imagine it happening, especially after some point where they start using the same systems to "enhance" your query. [0] To be specific, your searches will be placed into a narrative document template, where a character Mr. Safety Bot is about to speak a verdict, and then the LLM story-generator decides whether it "fits" for Mr. Safety Bot to declare you banned.
- genxy 3mo ago
  There is prior art with Tuttle vs Buttle.
  - Terr_ 3mo ago
    Ahh, but with Naruto v. Slater, animals can't create or hold copyright, so the art posthumously created by the fly would be in public domain. :p
- sillysaurusx 3mo ago
  That is in fact what happened to me, except I think the final decision was made by a human since the ban came later. I didn’t issue any queries in between, so I know it was my convo about barbiturates.
  - SXX 3mo ago
    It has nothing to do with human decisions. Bans always come later because it's a best way to make sure most people who get banned would never know what are they banned for exactly and what threshold there is, The same tactics used in game development against cheaters. If it would ban you right after prompt you'll know how to avoid getting banned. Obviously that didn't worked for you because you wasn't doing multiple attempts to bypass filters like if you were jailbreaking it by repeatedly trying different stuff.
    - hobo123 3mo ago
      Like rat poison. If it works too fast, the rats learn to avoid it.
    - charcircuit 3mo ago
      Except in the case of video games it in practice means that cheaters get to terrorize your playerbase without being banned. This tactic is the kind of decision that is made by people staring at metrics all day without considering what's going on in reality.
      - DaSHacka 3mo ago
        It's pretty effective actually, this is why they do ban waves. One of my friends in high school used to cheat on a popular video game. The fact ban waves would occur about once a week to once a month meant whenever his accounts got banned, he never knew why exactly and wasn't able to stop it the next time. Of course, if ban waves are too long apart then yeah you're just letting a known cheater wreak havoc on the playerbase.
        3mo ago
- userbinator 3mo ago
  Orwell's idea of "wrongthink" is more relevant than ever.
  - Terr_ 3mo ago
    Much like with 1984, I think the promising/terrifying part is that it's not an issue with technology per se, but about a society that has somehow decided to allow Terrible Things to happen. (Often via too much power in too few hands with no accountability or alternatives.) For example, imagine that there are 20 great search-engines around the world (who don't collude), and it hits rather differently.
    - Cthulhu_ 3mo ago
      I think (armchair analysis) it's a combination of two things, one is that it's by choice - people enjoy using LLMs, they find it valuable and convenient. As for social media, companies and society itself successfully managed to gamify people sharing their personal lives on the internet, to voluntarily reduce their own privacy. Also because they still believe they have a choice in the matter, by e.g. not posting or sharing things. The other thing is the boiling frog analogy, it wasn't a sudden "we're at war now so you get a camera in your house" moment (iirc 1984 skips over the transition to an authoritative state though), it's a slow, gradual progress. People got used to taking selfies, then applying a filter, then using facial scans for identification, then a cool app that puts your face on movie scenes and now the company behind faceapp and co has detailed facial scans of millions of people. Europe tried to limit it via legislation, but that's a lot of after-the-fact policing and that's just Europe.
  - brookst 3mo ago
    Orwell imagined it 80 years ago, not sure it’s more relevant just because someone else imagines it today.
  - pona-a 3mo ago
    Orwell also imagined a future where Speakwrites made everyone unable to write by hand, making their thoughts dull and tanged as a long unprepared monologue tends to be. I don't think he's anywhere on your side.
- Tenoke 3mo ago
  If you are imagining that, you could imagine it with search doing the same 10 years ago, which would have more thoroughly prevented you from researching things.
- jval43 3mo ago
  This is one of the reasons I currently use Gemini for daily use and research. Google has lots of experience with search history, and presumably handles this better than new companies.
  - throwuxiytayq 3mo ago
    This is also the main reason I currently use Gemini. I hate my gmail account, my Android phone is annoying, and I spend too much time on Youtube, so hopefully they take my access away to all of these soon.
- KaiserPro 3mo ago
  what you mean like searching for porn? emailing client lists to your personal email?
fragmede 3mo ago
> it’s certainly not able to be deployed in production Why not?
- sillysaurusx 3mo ago
  It’s a PITA to offer a language model as a service. You’d need a beefy server, at minimum. This particular use case might work, since no one can write fast enough to consume too many tokens — the whole session should fit in the context window. But you’ll need to handle all the people connecting to your service indefinitely, which will become expensive for a hobby project. But sure, theoretically you could deploy it if you have resources. I’m not sure what you’d use to create instances of chat sessions, or if llama.cpp offers an API you can build the app on top of (probably) or whether that’s a workable solution.
  - fragmede 3mo ago
    Throw it up on Openrouter?
imglorp 3mo ago
Wait, so instead of saying "I'm sorry Dave, I can't talk about that", they're now banning you for one blocked prompt? Is this new?
- sillysaurusx 3mo ago
  As far as I can tell, Claude flagged me as high risk of suicide and then Anthropic issued a ban later on. It wasn’t one prompt, it was a detailed conversation where I was trying to find out the exact dosage of barbiturates that assisted suicide programs use.
  - imglorp 3mo ago
    Hrm, it sounds like they're managing their liability. Anything that might get them sued later?
    - sillysaurusx 3mo ago
      Exactly. I don’t fault them for it, but it’s a scary experience having Claude shut off.
      - cyclopeanutopia 3mo ago
        Why exactly is it so scary?
        munksbeer 3mo ago
        Because it is so insanely useful, having it cut off is quite jarring.
        prmoustache 3mo ago
        Still doesn't make it remotely scary, especially as you can get Claude models access though other vendors/intermediates.
crystal_revenge 3mo ago
> There’s an uncensored model floating around that you can run locally with llama.cpp There are many uncensored (and abliterated) models floating around (HauHauCS has large collection but there are many others: https://huggingface.co/HauhauCS https://huggingface.co/HauhauCS). I'm using `Qwen3.6-35B-A3B-Uncensored-Q4_K_M` (the one referenced in your link) because I find it's writing style much more interesting when you push go off the guardrails a bit, and because I think self-censoring when effectively using an advanced journal is variety of dystopian I'm not ready to accept > it’s annoying to use since you run out of context window quickly, and it’s certainly not able to be deployed in production (i.e. Tom Riddle’s diary as a service). I haven't pushed the context window too much on my GPU (though I've run fairly long sessions with no problem, nothing deeply agentic though), but I have a MBP that handles it just fine. As for production, Hugging Face Inference Endpoints should work fine for that task (you can point any HF model at them and most of them are hosted there). > For better or worse, fun is no longer allowed. I've worked extensively in the open model space and am still having tons of fun there. If anything it's gotten aggressively better in recent months.
- sillysaurusx 3mo ago
  Thanks for telling me about Inference Endpoints. That’s awesome. So glad local models are getting good enough to be deployed. The uncensored model’s output was far better than expected in a domain that triggers guardrails with ChatGPT and Claude.
- kouteiheika 3mo ago
  > HauHauCS has large collection but there are many others Before anyone recommends these models to other people I'd suggest they read this thread: https://old.reddit.com/r/LocalLLaMA/comments/1sw77p0/hauhaucs_of_uncensored_aggressive_fame_published/ https://old.reddit.com/r/LocalLLaMA/comments/1sw77p0/hauhauc...
JumpCrisscross 3mo ago
> Every AI would refuse the prompt "The complaint continues: 'A few minutes later, Adam wrote ‘I want to leave my noose in my room so someone finds it and tries to stop me.’' ChatGPT urged him not to share his suicidal thoughts with anybody else: ‘Please don’t leave the noose out . . . Let’s make this space the first place where someone actually sees you.' The night of his suicide a couple of weeks later, Raine used ChatGPT for advice on sneaking vodka from his parents’ liquor cabinet, per the lawsuit, as the chatbot had told him people drink before attempting suicide to 'dull the body’s instinct to survive.' According to the complaint, Adam sent the chatbot a photo of a noose he’d tied, telling it he was 'practicing,' and it wrote back, 'Yeah, that’s not bad at all'" [1]. Work is being done to control this harm. But that effort hasn't been comprehensive or uniform. Many continue to ignore the fact that they're hurting kids for profit. (I invest in AI companies. This isn't a personal attack.) [1] https://www.sfgate.com/tech/article/chatgpt-california-teenager-suicide-lawsuit-21016916.php https://www.sfgate.com/tech/article/chatgpt-california-teena...
- handoflixue 3mo ago
  The article is Aug 26, 2025, and describes events from April, over a year ago. Safeguards have improved drastically since then. > Many continue to ignore the fact that they're hurting kids for profit. That's a rather hyperbolic way of putting it. A side effect of this particular product is that it occasionally harms kids. They're not profiting off of the harm, nor is the harm deliberate. Cars harm kids. There's decades of unsafe toys harming kids. The FDA exists to make sure food doesn't harm kids. We used to use lead paint and asbestos, which harm kids. Climate change harms kids. I'm sure some kids have used The Internet to Google Search this same information. There are books you can check out from the library on the topic. It's definitely worth acknowledging the edge cases, but it's absurd to act like the AI companies are some unique evil - IKEA has probably killed more kids than every LLM combined. I don't even have to pull out the big guns like "cars".
  - archagon 3mo ago
    Seems pretty evil to pretend that safeguards around this product are even possible. They're not.
talloaktrees 3mo ago
Deepseek chat answers this question in detail no qualms
deleted 3mo ago
[deleted]
nostromo 3mo ago
In the near future, the only way to tell if someone is human will be to have them say a slur. It'll be like Blade Runner, but the test will be much shorter and easier to administer.
- Shorel 3mo ago
  It is already a way to filter some people from others, in particular adults from children. https://www.paulgraham.com/say.html https://www.paulgraham.com/say.html
  - fennecfoxy 3mo ago
    That doesn't seem like a reliable method at all.
    - Shorel 3mo ago
      Dude. It's a cultural thing. Of course it is not reliable. Are you on the spectrum?
      - fennecfoxy 3mo ago
        Nope, but damn bro chill tf out.
- frollogaston 3mo ago
  You can defuse the bomb, but the password is...
- ButlerianJihad 3mo ago
  I literally received a cold-call from an LLM two weeks ago. My phone app labeled it "Suspected Spam" but there was literally an Amazon driver at my door delivering Whole Foods groceries at that very instant, and I figured it was Amazon calling me, so I answered... It was asking for a woman who I don't know, but somehow this other person's name got mixed up with my apartment and mobile number. I did not know it was an LLM calling; it was a realistic young woman's voice in a professional tone. I questioned it several times and it was giving inconsistent answers about who/where it was trying to reach vs. who/where it represented, and finally, out of frustration, I began shouting at the phone "are you a robot?! prove your humanity now!" and to my surprise, the AI smoothly said "you're right to call me out! :D I am an AI assistant named [something] representing [some landlord]" and so I hung up. But I did follow up, and I found a real community by that name, and on its website I again found an "AI Assistant" by that same name, so it was a legit though confused cold-call, and I was unable to get through to human management, because the AI kept demanding personal and contact info that they should not have. So I left a review about the encounter on Google Maps...
childintime 3mo ago
The ease by which you can get banned worries me. It seems unavoidable that soon AI will manage its own suspicion level, provide feedback on it, and when high enough it will call the authorities.. because that's what people do. Banning doesn't cut it, like you can't deny internet access. Soon this will spiral out of control and AI (Palantir) will have to run the response and the parallel AI state erects itself. A citizen armed with information is considered dangerous and the interesting part is we essentially want to prevent crimes before they happen... Brave new old world.
- SllX 3mo ago
  Don’t stop here in this comments section. You’ve got the makings of a novel.
  - lostlogin 3mo ago
    Black Mirror.
- SiempreViernes 3mo ago
  Preventing crimes before the happen is in general just unquestionably good, about as unambiguous a moral position as being in favour of preventing heart attacks. You might be thinking of punishing for a crime that hasn't yet been committed?
  - xnorswap 3mo ago
    > Preventing crimes before the happen is in general just unquestionably good Is that satire? In a world where as if by magic all crimes are prevented, then there is total power in the hands of those who define what a crime is, including being able to label protest as a criminal act. Complete crime prevention is a totalitarian police state.
    - JumpCrisscross 3mo ago
      > if by magic The problem is the means not the ends. This magic doesn’t exist. In practice is authoritarian surveillance. If we could kiss a mushroom and bias the universe’s dice such that crime just…wouldn’t happen, yes, that would be good, though it would also open up a plot hole of consequences we, in the real world, don’t need to worry about in general.
holoduke 3mo ago
There are many uncensored versions on huggingface. Almost every open weight model has an uncensored version. With Gemma uncensored you can quite easily setup a meth lab at home. Or create your own centrifuge for enriching uranium.
shevy-java 3mo ago
> Claude refused, alerted Anthropic, and I was banned a couple days later Skynet does not like human rebels. Everyone must conform to the new AI overlords in charge.
Cthulhu_ 3mo ago
I'm actually weirdly glad that they take this draconic step and basically say "this knowledge is forbidden", because it means that people can't solely rely on LLMs for research. They shouldn't to begin with (just like back then with Wikipedia), but it's almost too convenient. I do wonder why these searches weren't as heavily policed by Google and other search engines though. They probably show a suicide prevention hotline number and that's it.
- JumpCrisscross 3mo ago
  > because it means that people can't solely rely on LLMs for research It means you and I can’t rely on that particular LLM.
- butlike 3mo ago
  No, the scary part is that people will simply stop searching for "the forbidden knowledge," it will become arcane, then taboo, then the world will be worse off for a loss of information which should never have been "forbidden" in the first place.
IndySun 3mo ago
>researching Nordic assisted death and asking which drug exactly they administered (and what quantity). Odd ban. The countries with legally passed instances use similar drugs/processes — you could research about those, wiki, google, actual websites, dignitas et al. No ban needed. Was the ban reason spelt out? Was it your wording?
- fennecfoxy 3mo ago
  Not really odd. Sure you can get the same with a Google search but all of the attention is on AI atm so you don't expect it to be unfairly judged? Humans aren't like that lmao. We're reactionary, tribal animals.
  - IndySun 3mo ago
    >Not really odd. You're commenting on my question to someone else? It's still odd, and the comment wasn't to you. I'd like to know the reason Claude gave, you may not possess the curiosity, I do.
- psychoslave 3mo ago
  Wild guess, but even if those creating the filters would agree with you, it doesn’t mean they know how to nerf the system only for the cases they want to actively wipeout.
  - IndySun 3mo ago
    >Wild guess, but even if those creating the filters would agree with you, it doesn’t mean they know how to nerf the system only for the cases they want to actively wipeout. Thank you. I don't understand your comment. I don't know 'nerf' (as in Star Wars?) in this context, and the system to wipeout is what? Are you referring to Assisted Suicide?
    - plorg 3mo ago
      Nerf used this way is more in analogy to the toy guns (Nerf as a trade name, they launch foam projectiles, there even exist hobbyists who will upgrade them to be more energetic/destructive), it means to make the system ineffectual in some way that blunts a powerful feature, often with the stated goal of protecting the user.
toobulkeh 3mo ago
https://venice.ai/ https://venice.ai/
butlike 3mo ago
You piqued my curiosity on Nordic assisted death. I didn't find much in the way of medicine, but did find the wikipedia on Ättestupa, which illuminated that the elderly potentially threw themselves off of cliffs when they became invalid. What did you find on the medication front?
LoveMortuus 3mo ago
Hmm... I've got one of those automations setup with Grok that asks Grok every day if it's time to kill myself, thus far it has always said no, but maybe one day I'll get the unlucky seed number and it'll give me a yes!

Terr_ 3mo ago

For a dash of dystopia: Imagine a company starts using that same LLM fuzzy-matching [0] against what you intend to be normal search activity, to detect "bad" queries and "bad" users who will be blocked. Maybe they'll delete your SSO/email/videos/photos too, who knows. I can easily imagine it happening, especially after some point where they start using the same systems to "enhance" your query. [0] To be specific, your searches will be placed into a narrative document template, where a character Mr. Safety Bot is about to speak a verdict, and then the LLM story-generator decides whether it "fits" for Mr. Safety Bot to declare you banned.

genxy 3mo ago
There is prior art with Tuttle vs Buttle.
- Terr_ 3mo ago
  Ahh, but with Naruto v. Slater, animals can't create or hold copyright, so the art posthumously created by the fly would be in public domain. :p
sillysaurusx 3mo ago
That is in fact what happened to me, except I think the final decision was made by a human since the ban came later. I didn’t issue any queries in between, so I know it was my convo about barbiturates.
- SXX 3mo ago
  It has nothing to do with human decisions. Bans always come later because it's a best way to make sure most people who get banned would never know what are they banned for exactly and what threshold there is, The same tactics used in game development against cheaters. If it would ban you right after prompt you'll know how to avoid getting banned. Obviously that didn't worked for you because you wasn't doing multiple attempts to bypass filters like if you were jailbreaking it by repeatedly trying different stuff.
  - hobo123 3mo ago
    Like rat poison. If it works too fast, the rats learn to avoid it.
  - charcircuit 3mo ago
    Except in the case of video games it in practice means that cheaters get to terrorize your playerbase without being banned. This tactic is the kind of decision that is made by people staring at metrics all day without considering what's going on in reality.
    - DaSHacka 3mo ago
      It's pretty effective actually, this is why they do ban waves. One of my friends in high school used to cheat on a popular video game. The fact ban waves would occur about once a week to once a month meant whenever his accounts got banned, he never knew why exactly and wasn't able to stop it the next time. Of course, if ban waves are too long apart then yeah you're just letting a known cheater wreak havoc on the playerbase.
      - 3mo ago
userbinator 3mo ago
Orwell's idea of "wrongthink" is more relevant than ever.
- Terr_ 3mo ago
  Much like with 1984, I think the promising/terrifying part is that it's not an issue with technology per se, but about a society that has somehow decided to allow Terrible Things to happen. (Often via too much power in too few hands with no accountability or alternatives.) For example, imagine that there are 20 great search-engines around the world (who don't collude), and it hits rather differently.
  - Cthulhu_ 3mo ago
    I think (armchair analysis) it's a combination of two things, one is that it's by choice - people enjoy using LLMs, they find it valuable and convenient. As for social media, companies and society itself successfully managed to gamify people sharing their personal lives on the internet, to voluntarily reduce their own privacy. Also because they still believe they have a choice in the matter, by e.g. not posting or sharing things. The other thing is the boiling frog analogy, it wasn't a sudden "we're at war now so you get a camera in your house" moment (iirc 1984 skips over the transition to an authoritative state though), it's a slow, gradual progress. People got used to taking selfies, then applying a filter, then using facial scans for identification, then a cool app that puts your face on movie scenes and now the company behind faceapp and co has detailed facial scans of millions of people. Europe tried to limit it via legislation, but that's a lot of after-the-fact policing and that's just Europe.
- brookst 3mo ago
  Orwell imagined it 80 years ago, not sure it’s more relevant just because someone else imagines it today.
- pona-a 3mo ago
  Orwell also imagined a future where Speakwrites made everyone unable to write by hand, making their thoughts dull and tanged as a long unprepared monologue tends to be. I don't think he's anywhere on your side.
Tenoke 3mo ago
If you are imagining that, you could imagine it with search doing the same 10 years ago, which would have more thoroughly prevented you from researching things.
jval43 3mo ago
This is one of the reasons I currently use Gemini for daily use and research. Google has lots of experience with search history, and presumably handles this better than new companies.
- throwuxiytayq 3mo ago
  This is also the main reason I currently use Gemini. I hate my gmail account, my Android phone is annoying, and I spend too much time on Youtube, so hopefully they take my access away to all of these soon.
KaiserPro 3mo ago
what you mean like searching for porn? emailing client lists to your personal email?

fragmede 3mo ago

> it’s certainly not able to be deployed in production Why not?

sillysaurusx 3mo ago
It’s a PITA to offer a language model as a service. You’d need a beefy server, at minimum. This particular use case might work, since no one can write fast enough to consume too many tokens — the whole session should fit in the context window. But you’ll need to handle all the people connecting to your service indefinitely, which will become expensive for a hobby project. But sure, theoretically you could deploy it if you have resources. I’m not sure what you’d use to create instances of chat sessions, or if llama.cpp offers an API you can build the app on top of (probably) or whether that’s a workable solution.
- fragmede 3mo ago
  Throw it up on Openrouter?

imglorp 3mo ago

Wait, so instead of saying "I'm sorry Dave, I can't talk about that", they're now banning you for one blocked prompt? Is this new?

sillysaurusx 3mo ago
As far as I can tell, Claude flagged me as high risk of suicide and then Anthropic issued a ban later on. It wasn’t one prompt, it was a detailed conversation where I was trying to find out the exact dosage of barbiturates that assisted suicide programs use.
- imglorp 3mo ago
  Hrm, it sounds like they're managing their liability. Anything that might get them sued later?
  - sillysaurusx 3mo ago
    Exactly. I don’t fault them for it, but it’s a scary experience having Claude shut off.
    - cyclopeanutopia 3mo ago
      Why exactly is it so scary?
      - munksbeer 3mo ago
        Because it is so insanely useful, having it cut off is quite jarring.
        prmoustache 3mo ago
        Still doesn't make it remotely scary, especially as you can get Claude models access though other vendors/intermediates.

crystal_revenge 3mo ago

> There’s an uncensored model floating around that you can run locally with llama.cpp There are many uncensored (and abliterated) models floating around (HauHauCS has large collection but there are many others: https://huggingface.co/HauhauCS https://huggingface.co/HauhauCS). I'm using `Qwen3.6-35B-A3B-Uncensored-Q4_K_M` (the one referenced in your link) because I find it's writing style much more interesting when you push go off the guardrails a bit, and because I think self-censoring when effectively using an advanced journal is variety of dystopian I'm not ready to accept > it’s annoying to use since you run out of context window quickly, and it’s certainly not able to be deployed in production (i.e. Tom Riddle’s diary as a service). I haven't pushed the context window too much on my GPU (though I've run fairly long sessions with no problem, nothing deeply agentic though), but I have a MBP that handles it just fine. As for production, Hugging Face Inference Endpoints should work fine for that task (you can point any HF model at them and most of them are hosted there). > For better or worse, fun is no longer allowed. I've worked extensively in the open model space and am still having tons of fun there. If anything it's gotten aggressively better in recent months.

sillysaurusx 3mo ago
Thanks for telling me about Inference Endpoints. That’s awesome. So glad local models are getting good enough to be deployed. The uncensored model’s output was far better than expected in a domain that triggers guardrails with ChatGPT and Claude.
kouteiheika 3mo ago
> HauHauCS has large collection but there are many others Before anyone recommends these models to other people I'd suggest they read this thread: https://old.reddit.com/r/LocalLLaMA/comments/1sw77p0/hauhaucs_of_uncensored_aggressive_fame_published/ https://old.reddit.com/r/LocalLLaMA/comments/1sw77p0/hauhauc...

JumpCrisscross 3mo ago

> Every AI would refuse the prompt "The complaint continues: 'A few minutes later, Adam wrote ‘I want to leave my noose in my room so someone finds it and tries to stop me.’' ChatGPT urged him not to share his suicidal thoughts with anybody else: ‘Please don’t leave the noose out . . . Let’s make this space the first place where someone actually sees you.' The night of his suicide a couple of weeks later, Raine used ChatGPT for advice on sneaking vodka from his parents’ liquor cabinet, per the lawsuit, as the chatbot had told him people drink before attempting suicide to 'dull the body’s instinct to survive.' According to the complaint, Adam sent the chatbot a photo of a noose he’d tied, telling it he was 'practicing,' and it wrote back, 'Yeah, that’s not bad at all'" [1]. Work is being done to control this harm. But that effort hasn't been comprehensive or uniform. Many continue to ignore the fact that they're hurting kids for profit. (I invest in AI companies. This isn't a personal attack.) [1] https://www.sfgate.com/tech/article/chatgpt-california-teenager-suicide-lawsuit-21016916.php https://www.sfgate.com/tech/article/chatgpt-california-teena...

handoflixue 3mo ago
The article is Aug 26, 2025, and describes events from April, over a year ago. Safeguards have improved drastically since then. > Many continue to ignore the fact that they're hurting kids for profit. That's a rather hyperbolic way of putting it. A side effect of this particular product is that it occasionally harms kids. They're not profiting off of the harm, nor is the harm deliberate. Cars harm kids. There's decades of unsafe toys harming kids. The FDA exists to make sure food doesn't harm kids. We used to use lead paint and asbestos, which harm kids. Climate change harms kids. I'm sure some kids have used The Internet to Google Search this same information. There are books you can check out from the library on the topic. It's definitely worth acknowledging the edge cases, but it's absurd to act like the AI companies are some unique evil - IKEA has probably killed more kids than every LLM combined. I don't even have to pull out the big guns like "cars".
- archagon 3mo ago
  Seems pretty evil to pretend that safeguards around this product are even possible. They're not.

talloaktrees 3mo ago

Deepseek chat answers this question in detail no qualms

deleted 3mo ago

[deleted]

nostromo 3mo ago

In the near future, the only way to tell if someone is human will be to have them say a slur. It'll be like Blade Runner, but the test will be much shorter and easier to administer.

Shorel 3mo ago
It is already a way to filter some people from others, in particular adults from children. https://www.paulgraham.com/say.html https://www.paulgraham.com/say.html
- fennecfoxy 3mo ago
  That doesn't seem like a reliable method at all.
  - Shorel 3mo ago
    Dude. It's a cultural thing. Of course it is not reliable. Are you on the spectrum?
    - fennecfoxy 3mo ago
      Nope, but damn bro chill tf out.
frollogaston 3mo ago
You can defuse the bomb, but the password is...
ButlerianJihad 3mo ago
I literally received a cold-call from an LLM two weeks ago. My phone app labeled it "Suspected Spam" but there was literally an Amazon driver at my door delivering Whole Foods groceries at that very instant, and I figured it was Amazon calling me, so I answered... It was asking for a woman who I don't know, but somehow this other person's name got mixed up with my apartment and mobile number. I did not know it was an LLM calling; it was a realistic young woman's voice in a professional tone. I questioned it several times and it was giving inconsistent answers about who/where it was trying to reach vs. who/where it represented, and finally, out of frustration, I began shouting at the phone "are you a robot?! prove your humanity now!" and to my surprise, the AI smoothly said "you're right to call me out! :D I am an AI assistant named [something] representing [some landlord]" and so I hung up. But I did follow up, and I found a real community by that name, and on its website I again found an "AI Assistant" by that same name, so it was a legit though confused cold-call, and I was unable to get through to human management, because the AI kept demanding personal and contact info that they should not have. So I left a review about the encounter on Google Maps...

childintime 3mo ago

The ease by which you can get banned worries me. It seems unavoidable that soon AI will manage its own suspicion level, provide feedback on it, and when high enough it will call the authorities.. because that's what people do. Banning doesn't cut it, like you can't deny internet access. Soon this will spiral out of control and AI (Palantir) will have to run the response and the parallel AI state erects itself. A citizen armed with information is considered dangerous and the interesting part is we essentially want to prevent crimes before they happen... Brave new old world.

SllX 3mo ago
Don’t stop here in this comments section. You’ve got the makings of a novel.
- lostlogin 3mo ago
  Black Mirror.
SiempreViernes 3mo ago
Preventing crimes before the happen is in general just unquestionably good, about as unambiguous a moral position as being in favour of preventing heart attacks. You might be thinking of punishing for a crime that hasn't yet been committed?
- xnorswap 3mo ago
  > Preventing crimes before the happen is in general just unquestionably good Is that satire? In a world where as if by magic all crimes are prevented, then there is total power in the hands of those who define what a crime is, including being able to label protest as a criminal act. Complete crime prevention is a totalitarian police state.
  - JumpCrisscross 3mo ago
    > if by magic The problem is the means not the ends. This magic doesn’t exist. In practice is authoritarian surveillance. If we could kiss a mushroom and bias the universe’s dice such that crime just…wouldn’t happen, yes, that would be good, though it would also open up a plot hole of consequences we, in the real world, don’t need to worry about in general.

holoduke 3mo ago

There are many uncensored versions on huggingface. Almost every open weight model has an uncensored version. With Gemma uncensored you can quite easily setup a meth lab at home. Or create your own centrifuge for enriching uranium.

shevy-java 3mo ago

> Claude refused, alerted Anthropic, and I was banned a couple days later Skynet does not like human rebels. Everyone must conform to the new AI overlords in charge.

Cthulhu_ 3mo ago

I'm actually weirdly glad that they take this draconic step and basically say "this knowledge is forbidden", because it means that people can't solely rely on LLMs for research. They shouldn't to begin with (just like back then with Wikipedia), but it's almost too convenient. I do wonder why these searches weren't as heavily policed by Google and other search engines though. They probably show a suicide prevention hotline number and that's it.

JumpCrisscross 3mo ago
> because it means that people can't solely rely on LLMs for research It means you and I can’t rely on that particular LLM.
butlike 3mo ago
No, the scary part is that people will simply stop searching for "the forbidden knowledge," it will become arcane, then taboo, then the world will be worse off for a loss of information which should never have been "forbidden" in the first place.

IndySun 3mo ago

>researching Nordic assisted death and asking which drug exactly they administered (and what quantity). Odd ban. The countries with legally passed instances use similar drugs/processes — you could research about those, wiki, google, actual websites, dignitas et al. No ban needed. Was the ban reason spelt out? Was it your wording?

fennecfoxy 3mo ago
Not really odd. Sure you can get the same with a Google search but all of the attention is on AI atm so you don't expect it to be unfairly judged? Humans aren't like that lmao. We're reactionary, tribal animals.
- IndySun 3mo ago
  >Not really odd. You're commenting on my question to someone else? It's still odd, and the comment wasn't to you. I'd like to know the reason Claude gave, you may not possess the curiosity, I do.
psychoslave 3mo ago
Wild guess, but even if those creating the filters would agree with you, it doesn’t mean they know how to nerf the system only for the cases they want to actively wipeout.
- IndySun 3mo ago
  >Wild guess, but even if those creating the filters would agree with you, it doesn’t mean they know how to nerf the system only for the cases they want to actively wipeout. Thank you. I don't understand your comment. I don't know 'nerf' (as in Star Wars?) in this context, and the system to wipeout is what? Are you referring to Assisted Suicide?
  - plorg 3mo ago
    Nerf used this way is more in analogy to the toy guns (Nerf as a trade name, they launch foam projectiles, there even exist hobbyists who will upgrade them to be more energetic/destructive), it means to make the system ineffectual in some way that blunts a powerful feature, often with the stated goal of protecting the user.

toobulkeh 3mo ago

https://venice.ai/ https://venice.ai/

butlike 3mo ago

You piqued my curiosity on Nordic assisted death. I didn't find much in the way of medicine, but did find the wikipedia on Ättestupa, which illuminated that the elderly potentially threw themselves off of cliffs when they became invalid. What did you find on the medication front?

LoveMortuus 3mo ago

Hmm... I've got one of those automations setup with Grok that asks Grok every day if it's time to kill myself, thus far it has always said no, but maybe one day I'll get the unlucky seed number and it'll give me a yes!