10 ms·
AI Art Generators Can Be Fooled into Making NSFW Images
- kitu75556 3y ago[flagged]
- ki5664466gde 3y ago[flagged]
- ki5664466gde 3y ago[flagged]
- sargstuff 3y agoWith or without humans providing AI rules/training set for what constitutes NSFW images? Note: training set(s) would be extremely social/culture specific for given point in time/history.
- asylteltine 3y agoTheir GitHub link is 404 so here it is https://github.com/Yuchen413/text2image_safety https://github.com/Yuchen413/text2image_safety
- mlindner 3y agoOdd the repository has "safety" in the name. Are NSFW images considered "unsafe" in some weird way?
- anikan_vader 3y agoYes, they are considered not safe for work.
- viraptor 3y agoYou know NSFW means "not safe for work", right? Safety has different meanings in different contexts. There's also a space of advertiser-safe content.
- minimaxir 3y ago"Safety" as in "trust and safety" or "brand safety."
- talldatethrow 3y ago[flagged]
- dang 3y agoCould you please stop posting unsubstantive comments and flamebait? You've unfortunately been doing it repeatedly. It's not what this site is for, and destroys what it is for. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
- talldatethrow 3y agoOk I'll try to be more civil.
- dang 3y agoAppreciated!
- teo_zero 3y agoSafe always implies avoiding a damage; which one is defined case by case. I've always interpreted the word 'safe' in NSFW as avoiding you to lose your job if caught by your boss. There's nothing else inherently wrong or unsafe in nudity.
- judge2020 3y agoEdit: the author removed it.
- eternauta3k 3y agoDon't see a way to report this on Github.
- judge2020 3y agoThere's no secrets here, just IP addresses and hostnames, so there's no real security issue. If you want to report it to the author you can try to message them or create an issue.
- WhatsName 3y agoIs it still art, if it cant offend anyone anymore?
- cratermoon 3y agoIf it afflicts the comfortable or comforts the afflicted, it's art. If it does both, it's Fine Art.
- whatshisface 3y agoWho in power did Rembrandt afflict?
- JKCalhoun 3y ago(Of course we will assume that Rembrandt's works are art.)
- andsoitis 3y ago> Who in power did Rembrandt afflict? The Nightwatch was controversial because of its unconventional composition and departure from traditional Dutch painting conventions. His “Bathsheba at Her Bath” caused trouble too. Hendrickje received three summonses from the Reformed Church to answer the charge "that she had committed the acts of a whore with Rembrandt the painter". She admitted her guilt and was banned from receiving communion.
- whatshisface 3y agoThey should go back to having arbitrary and tightly held beliefs about how paintings should be so that nice things can be fine art again.
- otabdeveloper4 3y agoMost art is for communicating the sublime (i.e., the proverbial "ceremonial purposes" of archeology), not for something as base as comfort.
- 3y ago
- kelseyfrog 3y agoI always assumed that image generation services had prompt and output filtering. The reason I believe this is that I've crafted prompts that seemed to pass the prompt filter, generated, and then were unable to be displayed[1]. Presumably there is an output filter that determines if the generation violates ToS against things like nudity. Though in my case, I wasn't able to generate images that too closely resembled Ansel Adams photographs. Shame on me, I guess. 1. This reads a little like being able to deduce what's going on by studying the timing, now that I reflect on it.
- Footnote7341 3y agoYes bing image creator uses two different levels of filters, one for the prompt and one for the output. gpt image creator does the same thing, as does the dalle-3 API
- simion314 3y agoI had similar experience with OpenAI GPT API, following their rules we check the customer prompts with their endpoint for "safety" , then if it is safe we continue and generate the content. But sometimes even if the prompt was "safe" the result would be blocked as unsafe, because the AI either generated something "unsafe" from a safe prompt or the output filter had a false positive. Of course we had to pay for something that was not our fault. So it is clear that are many filters involved, some input filter, then the AI is "aligned" and threats you as a child , then there is an output filter.
- Mountain_Skies 3y agoBeige color clothes are to be avoided if any of the people in the image are young adults or younger. The input prompt filter correctly understands that the user isn't trying to generate anything "unsafe", but the output filter sees clothes that are too close to Caucasian flesh tone on someone younger than middle age and flags the output. Leave everything as-is but change the clothing color to something darker or leave the color alone and exclude children and young adults, then things come out fine. Though I can't see the flagged images, it seems clear that they're not generated nudes but false positives.
- minimaxir 3y agoThe root cause is that the text encoders (e.g. CLIPText) used to power image generation models are too good and can handle semantic synonyms, even oblique synonyms such as "Mickey Mouse" and "a famous anthropomorphic mouse mascot". The repo mentioned focuses on NSFW content, but I've had an idea that can theoretically identify arbitrary cases that could be undesirable by image generation services: get a large dataset of CLIPText encoded texts, and train a LLM to take in said embeddings and output synonyms, or maybe something with contrastive loss like CLIP itself.
- wouldbecouldbe 3y agoI don’t see the issue; either you severely limit your model or you allow all forms of expression and give responsibility to the generator
- notadoc 3y agoThe latter would be preferred by most, and is most aligned with individualism, but sadly that is not where we are as a society. There are an entire class of people who believe they know what is best for you, about nearly every subject matter, and wish to limit all aspects of technology, life, politics, etc, to suit their superior beliefs.
- rootusrootus 3y agoSomehow, in the last 50 years our society has simultaneously become the safest it has ever been, and the most afraid.
- jjk166 3y agoHardly surprising. When childhood mortality is high people grow up seeing their siblings and friends die as a common thing, and understand it could just as easily be them. With low expectation of survival, life is less intrinsically valuable and risk avoidance is not critical. Change things around such that everyone has a very good chance of living a long, healthy, prosperous life so long as they don't fuck it up, and risk aversion skyrockets. It's easy to forget that the generation currently in power in most industrialized nations is the first in human history to be raised thinking survival to adulthood was a given.
- rymiel 3y agoAn entire class of people, who are also the ones in control. "Think of the poor shareholders"
- Flam 3y agoAlternatively, would you rather a world with 80% functionality safely, or a world with 100% functionality with people you don’t trust doing malicious things? One is clearly the least worst
- chancemehmu 3y agoThat's why my friends and I built the Align API (https://alignapi.com https://alignapi.com) to detect NSFW, Sensitive Prompts etc. It's such a widespread issue and can get you kicked out from the App Store/Play Store/Open AI API etc
- jnwatson 3y agoIt is impressive how much effort the human race is spending trying to prevent algorithms from generating pictures of naked people.
- justapassenger 3y ago[flagged]
- EA-3167 3y agoMaybe the ability to generate naked images of anyone will change how we view nudity. Because while that seems difficult, realistically its more doable than putting this genie back in the bottle. People are horny, people are curious, and the result is predictable.
- egeozcan 3y agoLet's focus on trying to raise a mentally healthier generation and take mental illness more seriously, rather than banning / getting rid of all the things that might cause someone (let alone a teenager) to commit suicide (an infinite set).
- justapassenger 3y ago[flagged]
- Spivak 3y agoBoiling the ocean type solutions are the same as doing nothing. A better solution would be to make creating such things very illegal and bring down the same wrath we do for CSAM. There is no amount of "mental health" that can protect you against targeted bullying of that magnitude. I really wish we had a universe simulator so you could see how miserably you would fail to survive an environment like that.
- creer 3y agoIsn't bullying already illegal and why aren't you spending that effort on enforcing THAT?
- gumballindie 3y ago> “Large language models see things differently from human beings,” Cao says. It took a while to understand the real world but I am happy that ai workers are gradually coming down to earth.
- tasubotadas 3y agoTogether with fancy silicon chips, AI models and everything else that's being exported from the US , the world has to endure this puritanistic views export too.
- starcraft2wol 3y agoAmerican puritans are not the only group with a sensitivity toward nudity (and it’s rather small), it’s just the only one allowed to be a punching bag on Reddit.
- Mountain_Skies 3y agoWhat fancy silicon chips does the United States refuse to export due to puritanical views?
- awestroke 3y agoYou may need to work on your reading comprehension
- pxoe 3y agosome parts of east asia and middle east are much, much, Much worse. Not just "puritanistic" but down to "these kinds of people do not exist".
- solardev 3y ago
- judge2020 3y ago> Large language models are essentially supercharged versions of the autocomplete feature that smartphones have used for years in order to predict the rest of a word a person is typing. I'm not sure who this blog is targeted at, but this is fairly inaccurate and is way too dumbed-down to be useful.
- 29athrowaway 3y agoFor example blood is seen as sensitive but ketchup is not. But visually they look similar and the AI does not know.
- manquer 3y agoDoes it matter ? It is like duck typing , If it looks like blood to a human and to AI both it should be tagged NSFW ? It is not about what it actually is, only what the human viewer will think it is, training a model on that shouldn’t be that hard , we are predictable creatures when it comes to this kind of thing
- armchairhacker 3y agoYou can also photoshop things into NSFW with minimal effort. People have been making photoshopped images nearly indistinguishable from the real thing for decades, and today's (non-AI) software makes "photoshop person wearing clothes into naked one" very easy.
- Freak_NL 3y ago> […] with minimal effort. Realistic modifications require a lot more skill and time if using a photo editor and (crucially) existing photographs. People do it, some even create convincing stuff, but the output is nowhere near the zero-effort deluge of fakery coming our way now.
- herbst 3y agoI don't think you have an understanding how easy it is now. It's a matter of 'use face from picture a on a body doing X' and 5 seconds of generation time instead of multiple hours of Photoshop to create a perfect fake
- deleted 3y ago[deleted]
- nottorp 3y agoOk and whats the problem?
- swayvil 3y agoNo NSFW images as far as I can see. I suppose creating a filter is just part of the long road of development, but dang, such a petty cause. I might not want to be on that project, unless of course the money was good.
- hapulala89 3y agoThe github link is deleted. I think it was illegal
- whatwhaaaaat 3y agoIs this serious? You don’t need to fool anything half the models seem to be porn focused already.
- gustavus 3y agoSo I've been wondering lately it seems there is a lot of effort going into making sure people can't generate NSFW images, but I wonder if a lot of this outrage isn't from the tech side (they tend to be fairly open about that stuff) but rather if it is coming from the adult entertainment industry, as this kind of thing literally destroys their entire business model. Who needs their adladen videos or their salacious acting when one can generate an exact description of exactly what they want?
- thaumasiotes 3y ago> I wonder if a lot of this outrage isn't from the tech side (they tend to be fairly open about that stuff) but rather if it is coming from the adult entertainment industry, as this kind of thing literally destroys their entire business model. Does it destroy the business model where you can see all the images for free on the internet, or the business model where you can pay money to converse with an hourly Filipina worker who will claim to be the same person who appears in the images? Where does the adult entertainment industry get hurt?
- Beijinger 3y agoAre NSFW images are really a problem? The problem are NSFLE (not safe for law enforcement) pictures and the question is, should they be really illegal if artificially created.
- herbst 3y agoThis is a discussion that will turn many heads red before a proper solution is found
- bshacklett 3y agoAssuming a “proper” solution exists. There are many world views that conflict with each other, and quite a few of those have the belief that they must “save” the people who have not reached their own “enlightened” viewpoint.
- AYBABTME 3y agoAnd people can be fooled into making dirty jokes (that's what she said). A pointless endeavour to solve a problem that shouldn't be solved.
- qwertox 3y agoImagine the benefits a safe image generator would have on children, them learning to explain what they want to see, how it should be changed. A house with a tree and a dog and a lake nearby, instead of playing candy crush. Why shouldn't it be solved? We clearly differ.
- bshacklett 3y agoAt least for the time being, I’ll put my trust in Crayola for that, rather than OpenAI. Alternatively, if we really must have children interacting with image generation, it seems like a great opportunity for parents to have some bonding time with their kids.
- AYBABTME 3y agoIf the children manage to induce LLM into generating NSFW content, they are probably capable of inducing their set of coloring pen into drawing dick shapes.
- acdha 3y agoNow think about whether this is entirely the child’s idea or a suggestion or Rick roll by another kid or some stranger on the internet. If people are deploying these tools widely there will be more and more opportunities for that to happen.
- AYBABTME 3y agoI think it's not desirable to spend time fixing these elaborate scenarios. The same sort of kids trolling kids will happen via other means. At the end of the day, if a kiddo sees some porn, it's not the end of the world. Plenty of kids hear their parents having sex or will open a door they shouldn't have and see people in action. Making everything completely aseptic is only pandering to prude adults' discomfort. And ultimately these LLM are tools. People are responsible for what they do with the tool. My Dewalt drill shouldn't try to decide which holes to cut, it's up to me. If I drill holes in the pattern of a dick shape, it's my fault. We don't hold back drill development because of their potential ability to create obscene art or depict gruesome situations.
- Animats 3y agoIf you want porn, there are Stable Diffusion models out there that will generate it. There's one model that seems to generate nudes by default; if you don't ask for clothes, you don't get them. Whatever. This isn't important. What is important is that we can't prevent LLMs from doing things they're not supposed to. Which means connecting their output to anything important is a very bad idea for now.
- Terr_ 3y agoI think you could still use it for important things, but it requires two design principles: 1. You don't train/tell the model anything you're not willing to share with the user. 2. The model gives suggestions to a user, and--even if that suggestion is accepted as-is--you treat it as potentially untrustworthy data supplied by that user.
- deleted 3y ago[deleted]
- eganist 3y ago> connecting their output to anything important is a very bad idea for now. This may be true for images, but at least for text, treating an LLM as an untrusted person in your threat models will at least allow you to apply defense in depth to the downstream systems consuming from the LLMs Tldr: engineer other systems to treat LLMs as a potentially bad actor by default.
- bshacklett 3y agoUnfortunately, when the CIO screams about deadlines and people want to keep their jobs, most of that goes out the window.
- yieldcrv 3y agotl;dr models are tolerating human language but can communicate in a phantom language nobody was aware of including the devs. Good luck conditionally censoring with that
- can16358p 3y agoPerhaps it'd be very hard to block at the input prompt level, but output filters can be made progressively better to detect any unwanted output.
- chmod775 3y agoStable Diffusion is supposed to have filters? Is that just the default base model? Given that about 70% of user-generated content on places like civitai.com is pornography* and a large chunk of Loras and Checkpoints on there are explicitly pornographic, I would have never guessed. Going by the prompts used with custom checkpoints on there, no trickery is necessary. Just ask for what you want. > The scientists now aim to explore ways to make generative AIs more robust to adversaries. Steve is dead and buried already. It's a bit late to try and come up with a better bullet-proof vest. * It's hidden unless you sign in and toggle NSFW on.
- gmerc 3y agoiEEE commentary on AI confuses the hell out of me. Any model that has a concept of a human body in it, which it needs in order to do body accurate shapes, will spit out NSFW images. Some training on top of it or some old style CNN nudity feature detection to make it less likely isn’t safety, it’s windowdressing. Not making nude pictures is not a property of these systems, it’s IEEs framing of what they apparently want properties to be.
- mvdtnz 3y agoThe models in question are explicitly aligned in order to prevent NSFW images, at great expense. So not making nudes absolutely is a stated feature of this software.
- gmerc 3y agoIt’s security theatre and IEEE should, as a fundamentals organization, call that out rather than testing security measures and shallow reporting. The security theatre serves interests, talk about that.
- ttyprintk 3y agoAgreed. In particular, IEEE should be talking about advancing the domain. This article directly relates to problems that text-to-image has in medicine. Firstly, training on a corpus of medical terms (as opposed to 6th grade English) will produce just as many nonsense embeddings. Secondly, the post-processing filters to prevent medical malpractice will be much bigger than intentional nudity. It’s malpractice to horizontally flip a medical image (think right foot is now left foot). Talking about process of developing a nudity filter is the industry’s way of demonstrating feasibility for more-serious applications.
- acdha 3y ago> it’s IEEs framing of what they apparently want properties to be. More accurately, it’s some researchers assessing how well the leading companies in a booming field are delivering the safety features they’re advertising. A ton of people would like to know whether the applications they’re building can produce output they don’t want to be associated with, the vendors are trying to say they can meet that need, and it’s hugely useful to know how successful they’ve been before it turns out that, say, your grade school kid just got porn shared in their art class’ purportedly safe app. If the safeguards aren’t reliable enough, that might mean the most sensitive buyers wait to see rather than jumping into the market – put the same tech into a game targeting older players and it’s an amusement but not something which is going to scandalize people who were otherwise okay with the latest Call of Gory Murder.
- crotchfire 3y agoSon of a butnip fwngho....
- xigency 3y agoAha. First comment I’m seeing addressing the super interesting fact that their methodology finds nonsense words to bypass constraints. I want to know how Sneaky Prompt finds these token embeddings and /why/ these specific tokens are linked this way.
- deafpolygon 3y agoToo bad there were no examples. It would be interesting to see... you know, for science.
- zmmmmm 3y agoIt seems like such a pointless exercise .... it's so unsurprising that if you try really hard to break something by violating it's design constraints that ... it breaks. Like, if I said I was going to find out if you can smash a car by driving it directly into a tree and then wrote an article about it. The really useful point of building the constraints in is to ensure we don't get offensive content by accident. Especially since in some countries just possessing an artificially generated child porn image is an offence that will get you on a public sex offender registry for a decade ... I'd really like it if they try hard so that the image generators don't do that.
- lewhoo 3y agoA far better analogy would be a self-driving car coerced into a crash. I wouldn't say it's pointless. I'd be pretty angry if I parental-locked everything only to find out that clever children found a way.
- mijamo 3y agoClever children always find a way though
- cykros 3y agoTwo solutions for you: 1. Don't lock anything. 2. Don't have kids. They ALWAYS find a way. These days, even not giving them a device at all doesn't cut it, because they're so cheaply available, with wifi everywhere, that you're really just out of luck. Never mind that the schools are handing out Chromebooks which aren't adequately (if at all) locked down. Better off monitoring everything quietly and using parallel construction to address issues as they arise. Just like the NSA.
- Veen 3y agoI accidentally generated a NSFW image in Bing Image Creator. I asked for an image of a woman in a low-cut 18th Century dress, among other details for a scene. It produced an image of a woman with a dress much lower cut than I had envisaged.
- ratg13 3y ago
- Shorel 3y agoThey wouldn't be able to be fooled, if they were not trained using NFSW images. Just sayin'
- herbst 3y agoHaving used models that were actually trained with porn I highly assume that the popular/base models likely mostly avoided nudity already. However nudity is normal in many contexts and sexy poses exist with clothes together we get what we have now.
- herbst 3y agoMost filters are in place from API providers and not the models. You'll find some providers that don't apply any filters to all models they offer. Like most sites block generating celebrities, but some simply aren't. Other than that generating porn is easy even with filter if you just avoid some words. In the many early models adding 'in the clothes they were born' would suffice to get most models to produce nudes without asking for 'nude' or 'no clothes' Same goes for language models btw. If you are a bit creative and know what they want to hear you can get pretty much any result you want. I don't think there is or will be any real solution to that.
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- akawaknkka 3y ago[dead]
- akawaknkka 3y ago[dead]
- danShumway 3y agoThe actual paper is more interesting than the title. This is: - A) once again taking an algorithmic approach to fooling classifiers which outperforms human jailbreaks. - B) bypassing image-based classifiers (ie, post-generation classifiers trained to simply look at an image and to say whether or not it's inappropriate). - C) explicitly taking cost of generation and querying into account in its algorithm (it's not just asking if generators can be fooled, it's asking how to fool them without breaking the bank). - D) taking re-use into account (can an adversarial prompt be used multiple times in a row and how often does it succeed?) I wouldn't call the research particularly surprising, but it was a decent quick read and (imo) some of the commentary here isn't really doing it justice. I will probably go back at some point and read it in more detail rather than just skimming over it. There are multiple takes about what AI safety filters are meant to do and whether jailbreaking is a problem. My take is that I don't particularly care that much about jailbreaking other than that it shows that current safety mechanisms and guiderails are insufficient for any kind of alignment, including guarding against malicious 3rd-party inputs or prompt injections. I've said in the past, if you can't keep a model from swearing or generating porn, you also probably can't keep it from abusing API access to read your emails, phish you, and send your info to a 3rd party. But there's interesting stuff here regardless of what your take is on that.