7 ms·
Nightshade, a tool allowing artists to 'poison' AI models
- pixl97 3y agoPrevious discussion on topic https://news.ycombinator.com/item?id=37990750 https://news.ycombinator.com/item?id=37990750
- artninja1988 3y agoI wonder if you can build a classifier to detect those attacks by nightshade. Either way it's clear that these supply chain attacks need to be taken more seriously. I wonder how come humans aren't that prone to this adversarial nosie compared to models
- taeric 3y agoWhat makes you think we aren't as prone to this as the models are? There is a good reason folks rarely notice the many animals that are surrounding them on a walk in the woods. Even worse, we are prone to being influenced by the decor or surroundings in ways that we know, but still partake in. Mood lighting would be an easy example. And we are very influenced by our other senses at the same time as purely visual. (Though, I'm always amused by how much the VR experience of racing a car hits, all without too much haptic feedback from the steering wheel.)
- xyzal 3y agoPaper> https://arxiv.org/abs/2310.13828 https://arxiv.org/abs/2310.13828
- schleck8 3y agoThere was an attempt at poisoning training data with such invisible cloaking before and it turned out that this didn't really affect training quality (Glaze was the project name) Iirc there was also some license violation with the cloaking code ironically, it was like a pillow fight
- schleck8 3y agohttps://www.reddit.com/r/StableDiffusion/comments/11sqkh9/glaze_is_violating_gpl/ https://www.reddit.com/r/StableDiffusion/comments/11sqkh9/gl... Apparently this is the same team and they claim that the previous one was only intended to prevent training on an artist's style.
- brucethemoose2 3y agoThe ML community is stuffed with license violations. I have seen more than once instance of randomly copy-pasted code by sheer coincidence, recognizing it from older projects. ML researcher culture is hard to describe... But its something like "Everything is a research prototype, so why does it matter?"
- rhaps0dy 3y ago> There was an attempt at poisoning training data with such invisible cloaking before and it turned out that this didn't really affect training quality Source? The authors claim this is not the case, and they have experiments showing it.
- Garvi 3y agoThere's too much money at stake for the big players for this not to be labeled illegal any time now.
- NoboruWataya 3y agoWhich big players? There are some big players with an interest in facilitating the pervasion of AI, sure, but as far as I can tell there are are least as many big players who stand to lose from the widespread erosion of intellectual property rights that comes with it.
- Garvi 3y agoI had Microsoft, Google and Apple in mind as they are heavily invested into AI. Which big players are you talking about?
- NoboruWataya 3y agoPublishers, record labels, movie studios, basically all of the businesses that have spent decades lobbying to strengthen the intellectual property laws that are now being undermined.
- CelestialTeapot 3y agoYeah, but those guys want to get rid of those greedy artists, so they're all for AI they can leverage. https://arstechnica.com/ai/2023/08/google-record-labels-working-on-deal-covering-musical-deepfakes/ https://arstechnica.com/ai/2023/08/google-record-labels-work...
- jncfhnb 3y agoSeems incredibly unlikely
- Hamuko 3y agoIllegal where? Because I have a hard time believing it'd be illegal in the US due to the First Amendment existing.
- Devasta 3y agoDelighted to see it. Fuck AI art.
- schleck8 3y agoIt's the way forward though. AI won't stop like this, psychotherapy will be next in the coming years (look at Pi, it has a therapy mode now even). Just accept that AI will affect all aspects of life in 10 years
- timdiggerm 3y ago"forward" towards what?
- schleck8 3y agohttps://dictionary.cambridge.org/dictionary/english/forward https://dictionary.cambridge.org/dictionary/english/forward
- jncfhnb 3y agoMaking it easier to execute on creative ideas
- natch 3y agoToward the future and whatever it holds, is the usual meaning of that phrase. I think their point might be that we need to figure out how we live in a future with AI in it. Trying to deny AI with tools like this is not going to work.
- Auracle 3y agoTowards one man game devs being able to execute their vision, to middle class families being able to afford custom bespoke art, artists being able to greatly accelerate their workflows, etc. You know, technological progress. We don't stop it when it affects accountants, delivery drivers, programmers, etc. Art should be no different.
- klyrs 3y ago
- epups 3y agoSounds interesting, but without more details I'm not sure whether it would be so hard to detect and even correct for. Might be defeated even by some pixel averaging.
- XorNot 3y agoThese are temporary measures though. By most respects, training against data with this type of poisoning is going to improve the overall performance of models because they'll necessarily have to learn to disregard this sort of invisible data (which necessarily it's far a field from human perception).
- natch 3y agoWith a mere moment's thought, one thinks of the obvious way to defeat this: Simply rescale all images in a preprocessing step. Since this defeat is so easy, and so obvious, either the authors are daft, or they are brilliant and have thought of a way to prevent it. So, not wanting to waste much time on a paper from daft people, I did a quick skim of the paper. I don't see any mention of this defeat. No prevention of it, no assessment of it, not even any evidence of awareness of it. Did I miss something? I'd hate to see technically naive artist types (a mere subset of artist types, I acknowledge) being hoodwinked by such snake oil and paying money for such "tools."
- delecti 3y agoIf you rescale all your training data, then you're training on deliberately blurry images.
- byteknight 3y agoThat's not accurate. Downsizing doesn't always blur.
- natch 3y agoEver so slightly degraded perhaps in some cases but "blurry" is overstating it. And worth the trivial cost if such dumb-as-bricks techniques take hold. One could also submit each image to an image-to-image model with a blank prompt and the right settings to essentially ask for almost a copy, and come out with washed images. Could even do a sharpening (or upscale and sharpening for that matter) at the same time. It has a small (tiny really) GPU cost per image but it's a one-time cost.
- riidom 3y agoYou could use it to check if the image is poisoned. Original image is a "cat", downscaled/blurred/posterized or whatever is a "dog", then something is strange and better skip it.
- natch 3y agoI think “whatever” in your sentence is actually “have an infallibe human who knows all existent artistic styles tag it accurately with subject and style descriptions and then fine tune or train a model with it and then see if the model prompted with those descriptions produces images that match those descriptions.” So I think simply rescaling would be a lot easier.
- valine 3y agoThat sounds like a lot of work for something that can be worked around trivially. Add a small amount of random noise to the images and you’re done. Most people are probably doing this already for data augmentation purposes.
- NoMoreNicksLeft 3y agoSupposing this works as claimed, I'd be fine with denying copyright protection to any work proven to have the poison. Most likely though, it will be simple to strip out. Seriously, there's a "copyright derangement syndrome" going on right now, and these people need to settle the fuck down already.
- _chu1 3y agoI always had this idea in my head about feeding AI garbage it cannot regurgitate into anything useful but not a clue on how to create said garbage. Hopefully tools like this can stop image AI dead in it's tracks.
- xyzal 3y agoNot exactly concerning image AI, but regarding LLMs -- in case your site does not depend on indexing -- you can easily generate boatloads of garbage text via Markov chains and invisible-link them. One cool project for that> https://github.com/jsvine/markovify https://github.com/jsvine/markovify (Personally had the most fun outputs when constructing the corpus from technical articles mixed with esoteric babble) Active defense!
- michaelmrose 3y agoCan you explain why we would want to stop image AI in its tracks? Why do we want to replace taxi drivers and factory workers but not cheap crap commercial artists? Given that real art is both incredibly cheap to produce and incredibly cheap to distribute I should think those who are so inspired will continue to produce works to inspire us. I mean the advent of printers, cheap prints, and stock images didn't destroy art so why should AI?
- _chu1 3y agoArtists and photographers were never asked permission of their photos to be used to develop image AI yet they ended up doing so, it was built off theft and will continue to be build off theft by using user data from social media companies.
- michaelmrose 3y agoWhy is requiring their permission a good thing to start with?
- chankstein38 3y agoAh, someone who doesn't understand how this stuff works and is just unnecessarily mad. People posted that stuff on the internet, it's not retaining their property it's learning the nuances of it. So it's effectively me looking through someone's gallery then attempting to create something like their art. No, it doesn't create "perfect recreations" or "perfect stylized things of theirs" because the outputs frequently are only stylistically similar. I'm sure you could get something that looks like one of their original pieces out with some effort prompting and conditioning the output but haven't people been copying artwork for millenia? It's definitely not a new phenomenon and preventing people from using generative art isn't going to stop it. Think Alex Grey vs Cameron Gray. Both amazing artists but looking at Cameron's work he clearly is inspired by Alex. Did he steal that? Does he deserve to be shutdown? Maybe he asked permission but is Alex Grey financially burdened by Cameron's existence? Artists are just mad something can do cool stuff that allows people who haven't spent a decade practicing something to create what their minds imagine.