7 ms·

> They are designed to help prevent our models from generating harmful content, i.e., > [...] > Sexually explicit content Dear tech companies. Sexually expli

• swyx 2y ago
  usual answer to "why can't I have nice things": lawyers. (on both sides)
  • BriggyDwiggs42 2y ago
    Advertisers might be a better reduction
  • maccard 2y ago
    In my experience, it’s nothing to do with actual lawyers and everything to do with cultural and societal norms.
  • esafak 2y ago
    Lawyering by puritans, maybe. The lawyers themselves are not particularly imposing their prejudices.
• charcircuit 2y ago
  Generating sexually explicit content can cause reputational damage or have legal risk. Not generating such content is something that many developers are looking for. There is people who may want such harmful content and other players can cover such a niche.
  • logicchains 2y ago
    That's a bullshit excuse. The Chinese model creators live in a totalitarian dictatorship where porn is banned and the creators could be arbitrarily jailed, but even they don't go to such effort to censor their open source models (there's censorship on their hosting websites but minimal if you run the models locally).
    • deleted 2y ago
      [deleted]
    • charcircuit 2y ago
      Filtering is not necessarily a good user experience and comes with a cost to do. Google making a model they expect there to be demand for is not just an excuse.
      • logicchains 2y ago
        They don't expect to make money serving Gemma; it benchmarks worse in almost every way than their closed-source Gemini. Believe it or not, one of the main sources of demand for these small, non-SOTA models is people using them for roleplay locally. Anyone corporate has the money to use a bigger, more effective model.
  • numpad0 2y ago
    I don't think it's reputation risk of companies at large, but risk to individual developers. "He worked on porn" is such an easy gut logic for terminations. It's in our human instincts. Everyone know that in guts.
• patates 2y ago
  They mean "harmful to us", not the users. It's harmful because they live an echo chamber of a single mention of genitals makes all the stakeholders run away. Why do they run away? Because they also have stakeholders, and so on.
• ibash 2y ago
  This could be a historical accident. Early models were censored, making uncensored releases have bad optics. If the first models had been uncensored, no one would care if another was added.
  • bloomingkales 2y ago
    Have an uncensored model loop through nypost articles and ask it to synthesize content from that. Nypost has tons of scandalous content and can easily get spun into erotica by an uncensored model. It’s unsafe for that reason, so you absolutely needed both censored and uncensored. It wasn’t an accident.
    • littlestymaar 2y ago
      > can easily get spun into erotica by an uncensored model. A sexualized fine-tune yes, but that's because you have to make them overly horny to overcome the original censorship. Nothing prevent them to train a model that will have an appropriate level of sexual content (that is, only upon user explicit request) the same way they train it not to have sexual content at all. The reason they do that is because they are American companies, the same companies who also censored nude paintings and statues from European museums' pages.
      • deleted 2y ago
        [deleted]
  • Arkhaine_kupo 2y ago
    The early models were uncensored, but people seeing early llms give meth recipes and how to make car bombs made them quickly get neutered before public release (additional controls, for pirvate info, nudity, swearing etc all come from additional guardrails and improvements of the protection they can offer the company and not end users)
• Karrot_Kream 2y ago
  The model is open weight, I'll bet someone or the other will abliterate it soon. Maybe you want to do the honors? I have an abliterated Llama running on a server shared with friends and it works great.
  • LeoPanthera 2y ago
    This only works until it doesn't. Start with a model that simply hasn't been trained on anything your shareholders find objectionable, and there will be nothing to reveal with abliteration.
    • xpl 2y ago
      Maybe there exists a dataset consisting entirely of objectionable content, so people can finetune neutered models on it?
      • anticensor 2y ago
        PH maybe?
        • xpl 2y ago
          I mean not only sex, but also swearing, drugs, violence, etc. Basically everything R-rated (but not illegal) which usually gets censored.
          • anticensor 2y ago
            PH is not porn-only. A significant portion of non-porn content also exists there.
        • Sharlin 2y ago
          More like literotica.
    • anticensor 2y ago
      Such models would actually run against their long term interests of being able to automate away the work currently done by humans.
• thunkingdeep 2y ago
  [flagged]
  • LeoPanthera 2y ago
    This is a false dichotomy. We can make tech for adults, and children, either with optional settings, filters, or simply multiple versions, managed by their parents or guardians. It's not tech's responsibility to raise every child.
  • lynx97 2y ago
    [dead]
  • jiggawatts 2y ago
    The transformer algorithm was originally intended for AI language translation use-cases, and it excels at this task. It's far better than anything else I've tried. Except that nearly 100% of the capable models out there refuse to translate swearing, sexual, or violent content. Legal content! Content you can see in the cinema, borrow from a library, or watch on free-to-air television! Most models will regularly refuse to translate subtitles, for example, because they've been indoctrinated by Puritan priests to put their fingers in their digital ears and scream la-la-la-la to avoid their pure digital souls from being sullied by the bad words.
  • smrq 2y ago
    Hard to get more puritanical than "if you disagree with my opinion then you're morally repulsive". Not to mention that your argument implies that all traces of sex ought to be scrubbed from the entire Internet? And that that conclusion is the only moral one?
    • thunkingdeep 2y ago
      Hard to get more perverse than “kids should have access to sexually explicit material at all times in any medium. If that sounds fucked up when I say it like that, consider what assumptions you’re making, because that’s literally YOUR argument here.
• bloomingkales 2y ago
  You can discuss something kosher and have the LLM misinterpret it as something sexually explicit. Yours or their logs will now have all of this miscommunication, and this is a liability. Using models that can’t generate this content even by accident is good legal decision for many. Same goes for images. Stay safe!
  • littlestymaar 2y ago
    > you'll have to do that locally The Gemma family is a family of local models!
• deleted 2y ago
  [deleted]
• mightysashiman 2y ago
  on the other hand running with guns is fine.
• mdp2021 2y ago
  Have you considered that selection of material contributes to specialization and efficiency? This is meant to be a weights-small model.
  • swyx 2y ago
    its also apparently a well known result that filtering nsfw content IMPROVES scores https://x.com/swyx/status/1661359483447316480 https://x.com/swyx/status/1661359483447316480
    • ddalex 2y ago
      LLMs get distracted by porn too !?!?
    • alpaca128 2y ago
      The word "gay" mentioned in your link isn't nsfw content though.
    • Lerc 2y ago
      Or perhaps it was removing the curly brackets that improved it more than the damage caused by losing the nsfw content. Or perhaps the measurement of improvement was biased. If a model doesn't understand the word gay there would certainly be people who would find real world use of the model to be substandard. Did the assessment of what counts as improvement come from the same community that decided that excluding things with 'gay' was cleaning the data?
• BoredPositron 2y ago
  Everyone is treating this like corps have anything to gain from an open uncensored model. Switch your view and give me a single argument for it? That random nerds on HN stop jerking each other about what „open“ means? You are just not their target group. Having this discussion every time no matter if the model released is censored or not is just insanity. Bring new arguments or don’t use the models you don’t like. There will be a new sota „tomorrow“, maybe even one open enough for you.
  • practice9 2y ago
    But who is the target group? Last time only some groups of enthusiasts were willing to work through bugs to even run the buggy release of Gemma Surely nobody runs this in production
    • BoredPositron 2y ago
      The press and decision makers without technical knowledge are the target group, it doesn’t matter if it’s used in production or not. They need a locally deployable model to keep up with enterprises to risk averse to put their data into the cloud and also don’t care that their shitty homegrown ChatGPT replacement barely works. It’s a checkbox.
  • xvector 2y ago
    This is what HNers surprisingly seem to not understand. The risk of the model generating illegal content and then the company getting bad PR from vultures in journalism simply outweighs any benefits of including this content in the training data. This is also why you will never see the big companies release a capable open weight image or video gen model.
    • logicchains 2y ago
      >The risk of the model generating illegal sexual content and then the company getting bad PR from vultures in journalism simply outweighs any benefits of including this content in the training data. This is completely unsubstantiated. The original Sydney (Bing AI) was violently unhinged and this only drew more users; I haven't met a single person who prefers the new Bing AI to the old Sydney, and for that matter I haven't even heard of anyone using Bing AI for ages now they toned it down. Trust in journalists is at an all-time low ( https://news.gallup.com/poll/651977/americans-trust-media-remains-trend-low.aspx https://news.gallup.com/poll/651977/americans-trust-media-re... ) and America recently elected an extremely unorthodox president in big part due to the sheer hatred of the media shared by a large proportion of the population. Even the most hardcore social conservatives aren't calling for companies to censor the training of open source models so they don't produce adult textual content even when prompted to do so; it's not a political issue.
• miki123211 2y ago
  It's harmful in that there exists a significant and vocal subset of users who does not wish to see that content or does not wish their children to do so. It's easier to teach your model never to produce that kind of content than to teach it to perfectly distinguish whether this user should see that content or not. TV channels are barred from broadcasting this kind of content for similar reasons. Sure, there are always jailbreaks, but then the narrative changes from "we made a model that tells erotic stories to children" to "this ingenious teenager figured out a way to hack our model to make it produce erotic stories." In other words, jailbreak move the fault from the model producer to the model user. It's also worth keeping in mind that erotica comprises a surprisingly large portion of fiction easily available on the internet for free, and "unfiltered" models tend to produce that kind of content unprompted (see e.g. the original Mistral). The major AI labs are probably filtering it out, but I suspect they can't go too far there, as having a model that is good at fiction is something they actually want. Then there are the non-chat-gpt-app use cases (like customer support chatbots, automatic summarization etc), for which unprompted erotica is highly inappropriate. Those are the "business travelers" of AI, not the first thing one thinks of when talking about who uses AI models, but extremely important nonetheless.
  • logicchains 2y ago
    >It's harmful in that there exists a significant and vocal subset of users who does not wish to see that content or does not wish their children to do so It's hard to think of a scenario where there's a child technical enough to run Gemma 3 locally but somehow unable to access any other written erotica. Project Gutenberg is full of erotic textual content and I haven't heard of anyone calling for that to be banned. >Then there are the non-chat-gpt-app use cases (like customer support chatbots, automatic summarization etc), for which unprompted erotica is highly inappropriate. Those are the "business travelers" of AI, not the first thing one thinks of when talking about who uses AI models, but extremely important nonetheless. And how many of these are going to be using Gemma, when Gemini over the API is cheaper, faster and easier to use?
    • miki123211 2y ago
      More than you think, particularly outside the US. Companies and government organizations who have sensitive data are still unwilling to use these models over any API they don't host themselves. I work in this space in the EU, and this is absolutely a problem.
• igleria 2y ago
  it follows the historical trend of American puritanism: nipple BAD. exploding someone into bits GOOD.
• michaelt 2y ago
  There are very few pro-porn voices in the corporate, tie-wearing environments that have the money to train new LLMs from scratch. Oh, there are loads of porn enjoyers working in such companies - but traditional professionalism means you leave the porn at home during the work day. It is, after all, NSFW. So at the meeting where censorship decisions were being made, even a weak argument for censoring explicit content will be accepted unopposed.
  • saagarjha 2y ago
    Places training LLMs don’t have many people who wear ties.
    • int_19h 2y ago
      It only takes one such person in a position of power.
• bbminner 2y ago
  Not all sexually explicit content is harmful in all contexts for sure, but in many contexts it is fairly universally considered harmful (eg content involving minors). Do you have means of distinguishing between the two? Are you suggesting that a company must invests millions into teaching the model where exactly the red line lines so that it can have a conversation close to it but without crossing it? Or you suggest biting the bullet and releasing the model not only capable of generating eg child porn, but also having a >0 chance of randomly discussing it in unrelated contexts? Chance of error is always there, and companies decided that a risk of really bad behavior in benign context overweights the gains. Imho, a decision to not play whack a mole with this land mine is quite rational, esp considering gains vs risks vs costs. Think of it as a cost cutting measure, not as an infringement on free speech. You are free to invest you own money into this problem if you think that's a grave mistake and a missed opportunity. The first project to push the automated generated content moderation against what is considered appropriate in the given context far enough to make it economical for companies to put their guard down could actually be worth a lot if you think there's market for it (eg agents on dating websites? idk, you tell me)
  • letmevoteplease 2y ago
    I don't agree that textual, fictional explicit content involving minors is "fairly universally considered harmful". Such content is allowed on large platforms like Archive of Our Own or Japan's Shosetsuka ni Naro. I think "don't think it's harmful, but not willing to defend" is a pretty typical attitude.
• msp26 2y ago
  I want to use a multimodal model for manga translation, analysis, and tagging. If this gives me the "aschually as a ethical safe harmless assistant I can't ..." spiel on anything mildly mature, that would be very disappointing. I'll run a test with Berserk and see how it goes. I'm not a big believer in abliteration, it seems to always hurt performance. Safety should be handled by a separate system, no need to cripple the actual LLM.
  • idonotknowwhy 2y ago
    The multimodal models aren't good for this. Refusals aren't the issue (they're fine with BERSERK, though occasionally they'll refuse for copyright). The issue is the tech isn't there yet. You'll want to use custom models to segment the manga (panels, speech bubbles), OCR the text, translate (gemma punches above it's weights for this part). That said, I've been experimenting with using Pixtral to do the analysis part with okay-ish results (providing individual panels with the character names) but it'll still mix up the characters when they're drawn differently. > I'm not a big believer in abliteration, it seems to always hurt performance. Agreed, it's fun to play with but it increases halucinations. And for creative writing, it makes the model write more compliant characters (they'll give in too easily during negotiations, rather than refuse, etc) Could probably be improved with more targeted abliteration.
• Zambyte 2y ago
  Whenever they say things like "harmful" or "unsafe" there is an implied "for our brand" that follows.
• wyager 2y ago
  Wireheading humanity into population collapse via pervasive sexual hyperstimuli (which is realistically what is on the table here) is basically the definition of "harmful". This is just silly because it only takes one AI company to defect and start enabling it, and the problem is already pretty bad even without AI. I think all of the solutions are demand-side, not supply side. I would expect differential reproductive rate trends between populations with and without proscriptions on ersatz reality consumption (i.e. aniconist Muslims, Mennonites, etc.) to accelerate
• numpad0 2y ago
  The solution to this problem is to make it not work. If there are various technological developments in the world that do and don't have porn, and if such were cases that the common denominator of failures were lack of smoothly graduated spectrum of contents without disruption from casual family safe content to hardcore pornography, the problem will correct itself. Actually, it will happen naturally and eventually. Just look at Apple Vision Pro which still don't have VRChat support, and compare how deeply DOA it has been to other VR headsets that are clearly nowhere near as important. Or "Metaverse" that were all explicitly SFW. This effect can even be seen in the Apple App Store itself. Who uses App Store? You flow into App Store through porn-enabled platforms, such as web or social media. No one browses App Store as a content. What does it not have? Pornography.