10 ms·
Grok Sexual Images Draw Rebuke, France Flags Content as Illegal
- akutlay 9mo agoIt seems X's Grok became the first large LLM provider to weaken the content moderation rules. If people don't react enough, we will likely lose the first line of defense for keeping AI safe for anyone. Large providers need to act responsibly as the barrier of entry is practically 0.
- zajio1am 9mo agoThis is already possible, just download open-weight model and run it locally. It seems absurd to me to enforce content rules on AI services and even more that people on Hacker News advocate for that.
- nozzlegear 9mo agoWhy does that seem absurd to you?
- 7bit 9mo agoDon't feed the troll
- nutjob2 9mo agoSafety isn't just implemented via system prompts, it's also a matter of training and fine tuning, so what you're saying is incorrect. If you think people here think that models should enable CSAM you're out of your mind. There is such thing as reasonable safety, it not all or nothing. You also don't understand the diversity of opinion here. More broadly, if you don't reasonable regulate your own models and related work, then it attracts government regulation.
- zajio1am 9mo agoWhen these models are fine tuned to allow any kind of nudity, i would guess they also can be used to generate nude images of children. There is a level of generalization in these models. So it seems to me that arguing for restrictions that could be effectively implemented via prompt validation only is just indirect argumentation against open-weight models.
- chrisjj 9mo ago> When these models are fine tuned to allow any kind of nudity If you're suggesting Grok is fine-tuned to allow any kind of nudity, some evidence would be in order. The article suggests otherwise: "The service prohibits pornography involving real people’s likenesses and sexual content involving minors, which is illegal to create or distribute."
- Ancapistani 9mo agoI’ve run into “safeguards” far more frequently than I’ve actually tried to go outside the bounds of the acceptable use policy. For example, when I was attempting to use ChatGPT to translate a journal that was handwritten in Russian that contained descriptions of violent acts. I wasn’t generating violent content, much less advocating it - I was trying to understand something someone who had already committed a violent act had written. > If you think people here think that models should enable CSAM you're out of your mind. Intentional creation of “virtual” CSAM should be prosecuted aggressively. Note that that’s not the same thing as “models capable of producing CSAM”. I very much draw the line in terms of intent and/or result, not capability. > There is such thing as reasonable safety, it not all or nothing. You also don't understand the diversity of opinion here. I agree, but believe we are quite far away from “reasonable safety”, and far away from “reasonable safeguards”. I can get GPT-5 to try to talk me into committing suicide more easily than I can get it to translate objectionable text written in a language I don’t know.
- NedF 9mo ago[dead]
- wolvoleo 9mo agoTrue, CSAM should be blocked by all means. That's clear as day. However I think for Europe the regular sexual content moderation (even in text chat) is way over the top. I know the US is very prudish but here most people aren't. If you mention something erotic to a mainstream AI it will immediately close down which is super annoying because it blocks using it for such discussion topics. It feels a bit like foreign morals are being forced upon us. Limits on topics that aren't illegal should be selectable by the user. Not baked in hard to the most restricted standards. Similar to the way I can switch off safe search in Google. However CSAM generation should obviously be blocked and it's very illegal here too.
- sam_lowry_ 9mo agoFunnily Mistral is as much censored as ChatGPT. One should search Huggingface for role-playing models to have a decent level of erotic content, but even that does not guarantee you a pleasant experience.
- chrisjj 9mo agoSome misunderstanding here. This article makes abolutely no mention of CSAM. The objection is to "sexual content on X without people’s consent".
- johneth 9mo agoIt's nonconsensual generation of sexual content of real people that is breaking the law. And things like CSAM generation which is obviously illegal. > It feels a bit like foreign morals are being forced upon us. Welcome to the rest of the world, where US morals have been forced upon us for decades. You should probably get used to it.
- judahmeek 9mo agowhether it was the "first" definitely depends on your standards & focus: https://cloudsecurityalliance.org/blog/2025/02/19/deepseek-r1-ai-model-11x-more-likely-to-generate-harmful-content-security-research-finds https://cloudsecurityalliance.org/blog/2025/02/19/deepseek-r...
- akutlay 9mo agoAlso see: https://timesofindia.indiatimes.com/technology/tech-news/it-ministry-sends-letter-to-elon-musks-x-says-send-action-taken-report-on-indecent-content-on-the-platform/articleshow/126308260.cms https://timesofindia.indiatimes.com/technology/tech-news/it-...
- chrisjj 9mo ago“AI products must be tested rigorously before they go to market to ensure they do not have the capability to generate this material,” Not possible.
- SpicyLemonZest 9mo agoIt's extremely possible! As the source article notes, the Grok developers specifically chose to make their AI more permissive of sexual content than their competitors, which won't produce such images. This isn't a scenario where someone developed a complex jailbreak to circumvent Grok's built-in protections.
- ben_w 9mo ago> Not possible. To which governments, courts, and populations likely respond "We don't care if you can't go to market. We don't want models that do this. Solve it or don't offer your services here." Also… I think they probably could solve this. AI image analysis is a thing. AI that estimate age from an image has been a thing for ages. It's not like the idea of throwing the entire internet worth of images at a training sessions just to make a single "allowed/forbidden" filter is even ridiculous compared to the scale of all the other things going on right now.
- ls612 9mo agoThese models generate probably a billion images a day. If getting it wrong for even one of those images is enough to get the entire model banned then it probably isn't possible and this de facto outlaws all image models. That may precisely be the point of this tbh.
- lokar 9mo agoIf they can't prevent child porn, then it should be banned.
- ls612 9mo agoShould photoshop be outlawed? What about MS Paint? Both of them I’m pretty sure are capable of creating this stuff. Also, lets test your commitment to consistency on this matter. In most jurisdictions possession and creation of CSAM is a strict liability crime, so do you support prosecuting whatever journalist demonstrated this capability to the maximum extent of the law? Or are you only in favor of protecting children when it happens to advance other priorities of yours?
- ChrisArchitect 9mo agoEarlier: https://news.ycombinator.com/item?id=46460880 https://news.ycombinator.com/item?id=46460880 https://news.ycombinator.com/item?id=46466099 https://news.ycombinator.com/item?id=46466099 https://news.ycombinator.com/item?id=46468414 https://news.ycombinator.com/item?id=46468414
- josefritzishere 9mo agoIt would be Musk automating CSAM. This is how we're starting 2026?
- deleted 9mo ago[deleted]
- chrisjj 9mo agoThe article doesn't mention CSAM. It is about "created sexualized images of people including minors" and CSAM is not that.
- TheAlchemist 9mo agoWhat's amazing to me is that this is silenced by HN. It should be a major topic of discussion here.
- chrisjj 9mo agoWhat makes you say it is silenced by HN?
- JKCalhoun 9mo agoThis seems like it should be on the HN front page. And yesterday.
- chrisjj 9mo agoSurely it us missing just because many have flagged it. But that's far short of silencing it.
- JKCalhoun 9mo agoI imagine—but then that is just the HN community silencing it. Maybe flagging should have some different kind of weighting?
- chrisjj 9mo agoIt is not silenced.
- ndsipa_pomu 9mo agoI think discussions should be made immune from users flagging them if they have more than a certain amount of comments (50? 100?). If it's a problematic topic, then the administrator(s) should be able to flag it if they deem it necessary.
- chrisjj 9mo ago
- thrance 9mo agoGrok breaks France's hate-speech laws all the time but they're only going after it because it can create images of naked people? Musk's propaganda nexus should have been banned years ago here, but not for this stupid reason.
- johneth 9mo agoIt makes sexual images of real people without their consent. That's what's breaking the law.
- chrisjj 9mo agoIs an image of someone wearing only a bikini seriously claimed to be sexual here? Not by this article, for sure. "The service prohibits pornography involving real people’s likenesses and sexual content involving minors, which is illegal to create or distribute. Still, users have prompted Grok to digitally remove clothing from photos — mostly of women — so the subjects appeared to be wearing only underwear or bikinis."
- archagon 9mo agoTry doing that to your coworker and report back on how HR describes it in your offboarding meeting.
- chrisjj 9mo agoThe question is about an image not an action.
- tukarsdev 9mo agoRemoving people's clothes without their consent is assault, it doesn't matter if, in another setting, where they did consent to it, it would be fine. It obviously is sexual if you look at the intent of people doing it. Not the clothing itself.
- 9mo ago