9 ms·

How artists are sabotaging AI to take revenge on image generators

49 points by taimurkazmi 3y ago

taimurkazmi 3y ago
there is going to be a "pre-GPT" internet training set from 2022 and earlier that will basically be the last 'complete' not poisoned data set. As AI increases as an overall % of all online posts and activity it will death spiral on model quality.
- pupileater 3y ago
  Haha, like https://en.wikipedia.org/wiki/Low-background_steel https://en.wikipedia.org/wiki/Low-background_steel
  - tomrod 3y ago
    Not too far off! AI retraining on its own output is not a great thing.
    - GaggiX 3y ago
      ChatGPT trains on its own output by exploring its own sampling space using reinforcement learning, Dalle 3 was trained on a dataset of synthetic captions, they are both SOTA, you can use AI to train AI models.
    - danielbln 3y ago
      IT can be, though. High quality synthetic data for training is a thing, see Phi-2 [0] (and many more examples, people having been using GPT-4 outputs for finetuning a LOT to great avail). [0] https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/ https://www.microsoft.com/en-us/research/blog/phi-2-the-surp...
    - dragonwriter 3y ago
      > AI retraining on its own output is not a great thing. For most forms of GenAI, including image generators, using existing models to generate synthetic data (and then doing human labelling, etc., as appropriate) for fine tuning is quite common and important. Why do you think its not a great thing?
- sjfjsjdjwvwvc 3y ago
  I disagree. There is still plenty of genuine data being produced. Also in the case of images, most of them have digital watermarks. I see no reason to believe there will be a death spiral on model quality - in the last year or so the opposite has been happening. Of course the jury is still out on the future.
  - DaiPlusPlus 3y ago
    > Also in the case of images, most of them have digital watermarks. Do they now?
    - sjfjsjdjwvwvc 3y ago
      All the images you create with Stable Diffusion include the generation data in the PNG by default. Of course you can remove it.
      - dragonwriter 3y ago
        Moreover, anything created with StabilityAI's code in its default settings also includes an invisible watermark in the image data, not just the generation information in the metadata.
- amelius 3y ago
  Only if we're too lazy to pre-filter the content manually.
- dragonwriter 3y ago
  > there is going to be a "pre-GPT" internet training set from 2022 Well, yeah, there are several here, and I think all the major image generators are using some combination of them as their starting points: https://laion.ai/ https://laion.ai/ > As AI increases as an overall % of all online posts and activity it will death spiral on model quality. Nope, it will just mean that it will be more expensive to source additional training data on top of the massive trove of existing "clean" (from intentional poisoning) data (much of which isn't perfectly captioned and human work on improving captioning can improve its utility in model training, as can more advanced models with more advanced text encoders, etc.) If poisoning was widespread, it wouldn't impact "big model" quality much -- they aren't grabbing new random data on the internet for continuous training. It might drive up the expense of community fine tuning, which often does depend on sourcing representative imagery for target styles or concepts from, among other places, the internet.
mercurialsolo 3y ago
This is just a race to the bottom and creating artificial work - folks creating poisoned data sets and then armies of data engineers who look at creating valid data sets for building models that can be grounded.
sjfjsjdjwvwvc 3y ago
I feel for artists to an extent but no way are they going to succeed „ruining AI“ by poisoning image data sets. And even if that was feasible, what the hell do they gain? This is the whole „photography is not art“ or „digital image manipulation is not art“ or „oil painting is not art“ discussion all over again and I can believe serious artists actually take pet in this luddite theatre.
- chii 3y ago
  > And even if that was feasible, what the hell do they gain? a few more years of employment, at best. It sucks, but it's reality. There's no stopping the onslaught of AI. Better start learning how to take advantage of the AI tools to make yourself not irrelevant.
  - camillomiller 3y ago
    The lack of humanism in comments like this one are worrying. The fact that you don’t understand art or the artistic process doesn’t mean they are not relevant to a healthy society.
    - taylodl 3y ago
      Plus it ignores the fact that all of our jobs are going to go away to automation and AI. Which can be a utopia other than we humans seem to be hell-bent on creating dystopias.
      - sjfjsjdjwvwvc 3y ago
        I for one am looking forward to our fully automated gay space communist overlords.
    - chii 3y ago
      > they are not relevant to a healthy society. if they are relevant, then the proof would be that they'd be paid to create art by other humans who believe this notion of relevency.
kossTKR 3y ago
A nanoscopic amount of artists sabotaging a few images will not do anything to anyone, there'll just be a small version bump in vision models and the 0.0000001 contaminated images will also be classified correctly. While i really feel for the artist like everyone else potentially losing their livelihood to AI, this is an annoying fantasy story.
Exendroinient23 3y ago
Just wait for luddite political candidates becoming popular when white collar workers will start be inevitable replaced. It will be a total disaster for any sensible discourse.
- hospitalJail 3y ago
  I think the US already went through their populist cycle between 2008-2020. If that cycle theory is true, we shouldn't have to worry about it until our old days.
- ghusto 3y ago
  I dislike the unquestioned assumption that "Luddite bad, machine good". If you love progress for progress' sake, think about where you're progressing _to_. So far we've "progressed": - My furniture no longer being made by skilled carpenters: Everything is falls apart after a couple of years, and great, I have to put it together myself to begin with - Lowering of the bar to _literally anyone with a computer_ for music creation: I know all old people say this, so I won't break tradition: Your music is shite - Hi-Fi not being particularly hi-fi: A Bluetooth receiver playing lossy format crap out of terrible speakers - Cashiers to automatic checkouts: Cool, now I have to do their job without getting paid, then wait for the one staff member they kept on payroll to come and check my shopping and rescan everything because of course they don't trust me I could go on all night. Point is; progress just means forward motion, not always in a good direction.
bertil 3y ago
Poison works by being able to scale its impact dramatically: attacking chemical cornerstone processes, disabling a receptor, etc. ML is, to simplify, a statistical process that tries to identify probability distributions. That implies they’d have to purposefully degrade large amounts of images that the system has access to. Unless those images are impossible to distinguish from the others (because, say, they come from the same sites) I’m not sure how they’d do it consistently. They could mark sources, by using an unused combination of keywords and having a unique image attached to it, but poisoning—at scale? I’m not seeing a stable equilibrium where that’s possible.
deafpolygon 3y ago
Weird to phrase it as if the artists are in the wrong.
- masklinn 3y ago
  From the boot’s point of view, the skull is in the wrong.
- hospitalJail 3y ago
  Who is right? The people who are happy to have higher quality artwork at lower costs, allowing humanity to progress further? Or the establishment who want their own desires fulfilled at the expense of the rest of humanity?
  - sebastianconcpt 3y ago
    Can you explain how extinguishing the artists or, its practical equivalent, destroying the economic incentives for them to exist is being in the right?
    - hospitalJail 3y ago
      Seems like we are paying a guild.
      - sebastianconcpt 3y ago
        So?
bejd 3y ago
Previous discussions: https://news.ycombinator.com/item?id=38013151 https://news.ycombinator.com/item?id=38013151 https://news.ycombinator.com/item?id=37990750 https://news.ycombinator.com/item?id=37990750
beepbooptheory 3y ago
I agree with people this is a long shot, but how can you not admire the spunk? Whether we realize it or not yet, this is the kind of resolve and energy we need to bring to the future. Antagonism is not automatically luddite-ism! This is capitalism right? Dont we need the competition?
gcanyon 3y ago
We are on a roller coast ride of technological advancement. Gravity is pulling us inexorably forward, and we can't see where the track eventually goes. We have brakes, but not enough to stop forward movement entirely. We have some input on the direction we take, but our limited vision down the track makes it hard for us to anticipate and correct for any upcoming curves we don't like. It's going to be a bumpy ride, but hopefully we have fun along the way.
- taylodl 3y ago
  I think you misunderstand the violence that lies in our near future. Before, when we began automating manual labor we proclaimed you should get educated and design the machines and improve the processes. Now, we're developing tools to displace large swaths of those workers. There simply aren't other jobs for them to do that we haven't already automated. History shows this is not sustainable and will lead to protest and violence. I would suggest we avoid that, after all, that would be the smart thing to do, but our species has proved time and time again that it's not smart in the aggregate.
GaggiX 3y ago
It's never going to matter, ignoring that they are a extremely small percentage of images on internet, if they have enough signal to almost not be noticed by a human than there is more than enough signal to train an AI.
mysterypie 3y ago
> altering an image’s pixels in a way that wreaks havoc to computer vision but leaves the image unaltered to a human’s eyes Can someone who understands the original paper give an ELI5 on how that's possible? I understand how labeling an image of a cat as "dog" could poison a dataset. I also understand how adding images of a toaster (which sort of looks like a handbag) to a dataset of handbags could poison a dataset. But I don't understand what's happening in Figure 6 in the original paper. The pairs of pictures of the dogs, cars, etc. look absolutely identical. What exactly is happening there?
- taylorius 3y ago
  Computer vision systems are not as robust as human vision systems. There are subtle changes to pixel values that can utterly flummox a computer vision system, but seem undetectable to a human looking at the image.
  - gooob 3y ago
    what is the nature of these pixel changes?
    - taylorius 3y ago
      It varies. Specially crafted noise added to the pixel values. Looks like random noise, but obviously isn't. TBH I'm not an expert, but as I understand it, It is "trained" using the vision network, with a loss function that is some combination of being low amplitude, and reducing the strength of the correct image identification.
  - mysterypie 3y ago
    How do subtle pixel changes cause the image of a dog to classified as a cat rather than as gibberish?
    - amenhotep 3y ago
      This is a guess (I have read some of the paper but to me it does not seem to explain), but presumably you work backwards from the result you want. There is some vector that comes out of the classifier model that would represent cat, and another that would represent dog; you get the difference between these two models, and work back through the layers of matrix multiplication finding places where small perturbations in the input data force the classifier to make the exact errors you need.
ensocode 3y ago
Artists are happy to use the global working plattforms (Instagram, ...) to show and monetize their art to a world-wide audience but are unhappy if these content is used to train AI. If you show your art in just your own local gallery nobody will exploit it. Its the same in software engineering with public repos. So I guess we all have to adjust and use AI for the better instead of complaining and fighting it. Or am I missing the point here?
dig1 3y ago
This reminds me of an attempt when spammers tried (and still trying, but not as often as before) to poison antispam bayes filters with all sorts of garbage in email. The result is that filters became even more powerful over time.
sonicggg 3y ago
If they are fighting back, they are failing miserably, because the AI generators look better than ever. Midjourney, for example, is crazy good.
hospitalJail 3y ago
I talked to a luddite on this: "What if I take my own pictures, and train the data on my own pictures?" Their answer: "No" Luddites, nothing more. They will become obscure while everyone else advances. She uses her colored pencils, I use digital art. Guess who works at a coffee shop.
- camillomiller 3y ago
  These comments are so disheartening and disgusting. Are you enjoying your revenge of the nerds moment? Go fellate a robot, will ya
- ghusto 3y ago
  You say Luddite like it's a bad thing. > Guess who works at a coffee shop Her? Happy and fulfilled as a human being no doubt.
  - hospitalJail 3y ago
    Buddy she is depressed, in debt, and on a ton of pills.
hurrdurr56 3y ago
Solution: 1. Run a large number of verified unpoisoned images through the AI poisoning algorithm 2. Create a large high-quality data set consisting of the before and after poisoning image pairs 3. Train an AI using this data set so it can detect or even reverse the poisoning 4. AI train stops for no one
- Lockal 3y ago
  It probably can be simplified to writing an algorithm which detects that image was generated by AI (like https://deepmind.google/discover/blog/identifying-ai-generated-images-with-synthid/ https://deepmind.google/discover/blog/identifying-ai-generat...), and consider all AI-generated images as bad for training set by default.
dragonwriter 3y ago
Gotta give the team behind Nightshade and its earlier, rapidly debunked predecessor (Glaze) credit for a really strong media-relations operation. But that's about it, and this isn't meaningfully going to impact AI image generation at all.
Lockal 3y ago
It reminds me of a wave of comments "Deadline tomorrow !!! Everything you’ve ever posted becomes public from tomorrow. I give notice to Facebook it is strictly forbidden to disclose, copy, distribute, or take any other action against me. If you do not publish a statement at least once it will be tactically allowing the use of your photos, as well as the information contained in the profile status updates. DO NOT SHARE. Copy and paste." But now people are posting images instead of text. And it all looks so stupid that it only deals reputational blow to some subgroup of artists who now look like a group of technically illiterate luddites.

49 points by taimurkazmi 3y ago

there is going to be a "pre-GPT" internet training set from 2022 and earlier that will basically be the last 'complete' not poisoned data set. As AI increases as an overall % of all online posts and activity it will death spiral on model quality.

pupileater 3y ago
Haha, like https://en.wikipedia.org/wiki/Low-background_steel https://en.wikipedia.org/wiki/Low-background_steel
- tomrod 3y ago
  Not too far off! AI retraining on its own output is not a great thing.
  - GaggiX 3y ago
    ChatGPT trains on its own output by exploring its own sampling space using reinforcement learning, Dalle 3 was trained on a dataset of synthetic captions, they are both SOTA, you can use AI to train AI models.
  - danielbln 3y ago
    IT can be, though. High quality synthetic data for training is a thing, see Phi-2 [0] (and many more examples, people having been using GPT-4 outputs for finetuning a LOT to great avail). [0] https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/ https://www.microsoft.com/en-us/research/blog/phi-2-the-surp...
  - dragonwriter 3y ago
    > AI retraining on its own output is not a great thing. For most forms of GenAI, including image generators, using existing models to generate synthetic data (and then doing human labelling, etc., as appropriate) for fine tuning is quite common and important. Why do you think its not a great thing?
sjfjsjdjwvwvc 3y ago
I disagree. There is still plenty of genuine data being produced. Also in the case of images, most of them have digital watermarks. I see no reason to believe there will be a death spiral on model quality - in the last year or so the opposite has been happening. Of course the jury is still out on the future.
- DaiPlusPlus 3y ago
  > Also in the case of images, most of them have digital watermarks. Do they now?
  - sjfjsjdjwvwvc 3y ago
    All the images you create with Stable Diffusion include the generation data in the PNG by default. Of course you can remove it.
    - dragonwriter 3y ago
      Moreover, anything created with StabilityAI's code in its default settings also includes an invisible watermark in the image data, not just the generation information in the metadata.
amelius 3y ago
Only if we're too lazy to pre-filter the content manually.
dragonwriter 3y ago
> there is going to be a "pre-GPT" internet training set from 2022 Well, yeah, there are several here, and I think all the major image generators are using some combination of them as their starting points: https://laion.ai/ https://laion.ai/ > As AI increases as an overall % of all online posts and activity it will death spiral on model quality. Nope, it will just mean that it will be more expensive to source additional training data on top of the massive trove of existing "clean" (from intentional poisoning) data (much of which isn't perfectly captioned and human work on improving captioning can improve its utility in model training, as can more advanced models with more advanced text encoders, etc.) If poisoning was widespread, it wouldn't impact "big model" quality much -- they aren't grabbing new random data on the internet for continuous training. It might drive up the expense of community fine tuning, which often does depend on sourcing representative imagery for target styles or concepts from, among other places, the internet.

mercurialsolo 3y ago

This is just a race to the bottom and creating artificial work - folks creating poisoned data sets and then armies of data engineers who look at creating valid data sets for building models that can be grounded.

sjfjsjdjwvwvc 3y ago

I feel for artists to an extent but no way are they going to succeed „ruining AI“ by poisoning image data sets. And even if that was feasible, what the hell do they gain? This is the whole „photography is not art“ or „digital image manipulation is not art“ or „oil painting is not art“ discussion all over again and I can believe serious artists actually take pet in this luddite theatre.

chii 3y ago
> And even if that was feasible, what the hell do they gain? a few more years of employment, at best. It sucks, but it's reality. There's no stopping the onslaught of AI. Better start learning how to take advantage of the AI tools to make yourself not irrelevant.
- camillomiller 3y ago
  The lack of humanism in comments like this one are worrying. The fact that you don’t understand art or the artistic process doesn’t mean they are not relevant to a healthy society.
  - taylodl 3y ago
    Plus it ignores the fact that all of our jobs are going to go away to automation and AI. Which can be a utopia other than we humans seem to be hell-bent on creating dystopias.
    - sjfjsjdjwvwvc 3y ago
      I for one am looking forward to our fully automated gay space communist overlords.
  - chii 3y ago
    > they are not relevant to a healthy society. if they are relevant, then the proof would be that they'd be paid to create art by other humans who believe this notion of relevency.

kossTKR 3y ago

A nanoscopic amount of artists sabotaging a few images will not do anything to anyone, there'll just be a small version bump in vision models and the 0.0000001 contaminated images will also be classified correctly. While i really feel for the artist like everyone else potentially losing their livelihood to AI, this is an annoying fantasy story.

Exendroinient23 3y ago

Just wait for luddite political candidates becoming popular when white collar workers will start be inevitable replaced. It will be a total disaster for any sensible discourse.

hospitalJail 3y ago
I think the US already went through their populist cycle between 2008-2020. If that cycle theory is true, we shouldn't have to worry about it until our old days.
ghusto 3y ago
I dislike the unquestioned assumption that "Luddite bad, machine good". If you love progress for progress' sake, think about where you're progressing _to_. So far we've "progressed": - My furniture no longer being made by skilled carpenters: Everything is falls apart after a couple of years, and great, I have to put it together myself to begin with - Lowering of the bar to _literally anyone with a computer_ for music creation: I know all old people say this, so I won't break tradition: Your music is shite - Hi-Fi not being particularly hi-fi: A Bluetooth receiver playing lossy format crap out of terrible speakers - Cashiers to automatic checkouts: Cool, now I have to do their job without getting paid, then wait for the one staff member they kept on payroll to come and check my shopping and rescan everything because of course they don't trust me I could go on all night. Point is; progress just means forward motion, not always in a good direction.

bertil 3y ago

Poison works by being able to scale its impact dramatically: attacking chemical cornerstone processes, disabling a receptor, etc. ML is, to simplify, a statistical process that tries to identify probability distributions. That implies they’d have to purposefully degrade large amounts of images that the system has access to. Unless those images are impossible to distinguish from the others (because, say, they come from the same sites) I’m not sure how they’d do it consistently. They could mark sources, by using an unused combination of keywords and having a unique image attached to it, but poisoning—at scale? I’m not seeing a stable equilibrium where that’s possible.

deafpolygon 3y ago

Weird to phrase it as if the artists are in the wrong.

masklinn 3y ago
From the boot’s point of view, the skull is in the wrong.
hospitalJail 3y ago
Who is right? The people who are happy to have higher quality artwork at lower costs, allowing humanity to progress further? Or the establishment who want their own desires fulfilled at the expense of the rest of humanity?
- sebastianconcpt 3y ago
  Can you explain how extinguishing the artists or, its practical equivalent, destroying the economic incentives for them to exist is being in the right?
  - hospitalJail 3y ago
    Seems like we are paying a guild.
    - sebastianconcpt 3y ago
      So?

bejd 3y ago

Previous discussions: https://news.ycombinator.com/item?id=38013151 https://news.ycombinator.com/item?id=38013151 https://news.ycombinator.com/item?id=37990750 https://news.ycombinator.com/item?id=37990750

beepbooptheory 3y ago

I agree with people this is a long shot, but how can you not admire the spunk? Whether we realize it or not yet, this is the kind of resolve and energy we need to bring to the future. Antagonism is not automatically luddite-ism! This is capitalism right? Dont we need the competition?

gcanyon 3y ago

We are on a roller coast ride of technological advancement. Gravity is pulling us inexorably forward, and we can't see where the track eventually goes. We have brakes, but not enough to stop forward movement entirely. We have some input on the direction we take, but our limited vision down the track makes it hard for us to anticipate and correct for any upcoming curves we don't like. It's going to be a bumpy ride, but hopefully we have fun along the way.

taylodl 3y ago
I think you misunderstand the violence that lies in our near future. Before, when we began automating manual labor we proclaimed you should get educated and design the machines and improve the processes. Now, we're developing tools to displace large swaths of those workers. There simply aren't other jobs for them to do that we haven't already automated. History shows this is not sustainable and will lead to protest and violence. I would suggest we avoid that, after all, that would be the smart thing to do, but our species has proved time and time again that it's not smart in the aggregate.

GaggiX 3y ago

It's never going to matter, ignoring that they are a extremely small percentage of images on internet, if they have enough signal to almost not be noticed by a human than there is more than enough signal to train an AI.

mysterypie 3y ago

> altering an image’s pixels in a way that wreaks havoc to computer vision but leaves the image unaltered to a human’s eyes Can someone who understands the original paper give an ELI5 on how that's possible? I understand how labeling an image of a cat as "dog" could poison a dataset. I also understand how adding images of a toaster (which sort of looks like a handbag) to a dataset of handbags could poison a dataset. But I don't understand what's happening in Figure 6 in the original paper. The pairs of pictures of the dogs, cars, etc. look absolutely identical. What exactly is happening there?

taylorius 3y ago
Computer vision systems are not as robust as human vision systems. There are subtle changes to pixel values that can utterly flummox a computer vision system, but seem undetectable to a human looking at the image.
- gooob 3y ago
  what is the nature of these pixel changes?
  - taylorius 3y ago
    It varies. Specially crafted noise added to the pixel values. Looks like random noise, but obviously isn't. TBH I'm not an expert, but as I understand it, It is "trained" using the vision network, with a loss function that is some combination of being low amplitude, and reducing the strength of the correct image identification.
- mysterypie 3y ago
  How do subtle pixel changes cause the image of a dog to classified as a cat rather than as gibberish?
  - amenhotep 3y ago
    This is a guess (I have read some of the paper but to me it does not seem to explain), but presumably you work backwards from the result you want. There is some vector that comes out of the classifier model that would represent cat, and another that would represent dog; you get the difference between these two models, and work back through the layers of matrix multiplication finding places where small perturbations in the input data force the classifier to make the exact errors you need.

ensocode 3y ago

Artists are happy to use the global working plattforms (Instagram, ...) to show and monetize their art to a world-wide audience but are unhappy if these content is used to train AI. If you show your art in just your own local gallery nobody will exploit it. Its the same in software engineering with public repos. So I guess we all have to adjust and use AI for the better instead of complaining and fighting it. Or am I missing the point here?

dig1 3y ago

This reminds me of an attempt when spammers tried (and still trying, but not as often as before) to poison antispam bayes filters with all sorts of garbage in email. The result is that filters became even more powerful over time.

sonicggg 3y ago

If they are fighting back, they are failing miserably, because the AI generators look better than ever. Midjourney, for example, is crazy good.

hospitalJail 3y ago

I talked to a luddite on this: "What if I take my own pictures, and train the data on my own pictures?" Their answer: "No" Luddites, nothing more. They will become obscure while everyone else advances. She uses her colored pencils, I use digital art. Guess who works at a coffee shop.

camillomiller 3y ago
These comments are so disheartening and disgusting. Are you enjoying your revenge of the nerds moment? Go fellate a robot, will ya
ghusto 3y ago
You say Luddite like it's a bad thing. > Guess who works at a coffee shop Her? Happy and fulfilled as a human being no doubt.
- hospitalJail 3y ago
  Buddy she is depressed, in debt, and on a ton of pills.

hurrdurr56 3y ago

Solution: 1. Run a large number of verified unpoisoned images through the AI poisoning algorithm 2. Create a large high-quality data set consisting of the before and after poisoning image pairs 3. Train an AI using this data set so it can detect or even reverse the poisoning 4. AI train stops for no one

Lockal 3y ago
It probably can be simplified to writing an algorithm which detects that image was generated by AI (like https://deepmind.google/discover/blog/identifying-ai-generated-images-with-synthid/ https://deepmind.google/discover/blog/identifying-ai-generat...), and consider all AI-generated images as bad for training set by default.

dragonwriter 3y ago

Gotta give the team behind Nightshade and its earlier, rapidly debunked predecessor (Glaze) credit for a really strong media-relations operation. But that's about it, and this isn't meaningfully going to impact AI image generation at all.

Lockal 3y ago

It reminds me of a wave of comments "Deadline tomorrow !!! Everything you’ve ever posted becomes public from tomorrow. I give notice to Facebook it is strictly forbidden to disclose, copy, distribute, or take any other action against me. If you do not publish a statement at least once it will be tactically allowing the use of your photos, as well as the information contained in the profile status updates. DO NOT SHARE. Copy and paste." But now people are posting images instead of text. And it all looks so stupid that it only deals reputational blow to some subgroup of artists who now look like a group of technically illiterate luddites.