4 ms·

This similar thing was posted a few weeks ago, and also apparently two years ago, glaze also from uchicago https://news.ycombinator.com/item?id=46364338 https:

by andy99 9mo ago

This similar thing was posted a few weeks ago, and also apparently two years ago, glaze also from uchicago https://news.ycombinator.com/item?id=46364338 https://news.ycombinator.com/item?id=46364338 https://news.ycombinator.com/item?id=35224219 https://news.ycombinator.com/item?id=35224219 We’ve seen this arms race before and know who wins. It’s all snake oil imo

vidarh 9mo ago
> We’ve seen this arms race before and know who wins. It’s all snake oil imo It's kinda funny in a way because effectively they're helping iron out ways in which these models "see" differently to humans. Every escalation will in the end just help make the models more robust... That they are disclosing the tools rather than e.g. creating a network service makes this even easier.
- jappgar 9mo ago
  And now you know the only reason these labs get any funding. It's all to benefit industry, whether the academics realize it or not.
tgv 9mo ago
Idk. Perhaps this technique doesn't work, but if someone comes up with a working system, and LLMs start using techniques to counter it, artists might have a leg to stand upon, as the use of the counter-technique makes clear that the scraper never had any intention of respecting terms of use.
- vidarh 9mo ago
  They won't need to use counter techniques beyond fixing incorrect output from their models by making the general training methods more robust to features not seen by humans.
- pixl97 9mo ago
  No, not really. In fact I would say the opposite is true. LLMs must protect against this as a security measure in unified models or things the LLM 'sees' may be faked. If for example someone could trick you into seeing a $1 bill as a $10 it would be considered a huge failure on your part and it would be trained out of you if you wanted to remain employed.
YeGoblynQueenne 9mo ago
>> We’ve seen this arms race before and know who wins. It’s all snake oil imo I haven't and I don't know who wins. Who wins? Adversarial examples aren't snake oil, if that's what you meant. There's a rich literature on both producing and bypassing them that has accumulated over the years, but while I haven't kept abreast with it, my recollection is that the bottom line is like that for online security: there's never a good reason not to make sure your system is up to date and protected from attacks, even if there exist attacks that can bypass any defense. Where in this case attack and defense can both describe what artists want to do with their work.
- jappgar 9mo ago
  In an arms race, the party with the most money always wins.
  - gspr 9mo ago
    Citation needed.
    - deleted 9mo ago
      [deleted]
- torginus 9mo ago
  Aren't adversarial examples have to be trained to be effective against a specific recognizer? I could imagine you could make one that was effective against multiple recognizers, but not in general. I'd also guess it'd be easy to get rid of this vulnerability on the model side.
- pixl97 9mo ago
  This isn’t security... Don't confuse attempting to make AI misclassify an image as a security measure. And yes, this is snake oil and the AI wins every time. At the end of the day a human has to be able to interpret the image, and I'd add another constraint of not thinking it looks ugly. This puts a very hard floor on what a poisoner can put in an image before the human gets sick. In a rapid turn around GAN you hit that noise floor really quickly.
oth001 9mo ago
Doesn't mean artists should make it easy for these AI companies to steal artist IP. It doesn't take long to do and seems effective enough from what I've seen. BTW This is how cybersecurity works (cat and mouse etc)
- vidarh 9mo ago
  The problem is that it is an inherently intractable problem with the (temporary) solution space shrinking with each mitigation, as the images still needs to look good to people.
  - pixl97 9mo ago
    Exactly. This isn't like encryption where you can just keep adding more bits. Every iteration that gets closer to simulating how people see sets the floor.
- jappgar 9mo ago
  Real security systems don't publicize how they work. This is just grandstanding. Half the people from this lab will go on to work for AI companies.
  - daeken 9mo ago
    > Real security systems don't publicize how they work. 175 years of history would disagree with you: https://en.wikipedia.org/wiki/Security_through_obscurity https://en.wikipedia.org/wiki/Security_through_obscurity
    - jappgar 9mo ago
      That old saw. Downvote all you want. Adversarial engineering does indeed rely on obscurity, they just don't tell you that.
      - daeken 9mo ago
        I've been working in security for more than 20 years and have seen the deleterious effects of security through obscurity first-hand. Why does "adversarial engineering" rely on obscurity?
zelphirkalt 9mo ago
Isn't there a huge cost imbalance? As in easy to add some noise, difficult to remove reliably, so that even if it gets removed, it could still be counted as a partial win defending against unwanted AI scraping.
cmxch 9mo ago
AI model makers win, luddites lose. Never mind that the more people try to corrupt a model, the more likely that future models will catch these corruption attempts as security and trust/safety issues to fix and work around. The next Nightshade will eventually be viewed as malware to a model and then worked around, reconstructing around the attempt to break a model.

by andy99 9mo ago

vidarh 9mo ago

> We’ve seen this arms race before and know who wins. It’s all snake oil imo It's kinda funny in a way because effectively they're helping iron out ways in which these models "see" differently to humans. Every escalation will in the end just help make the models more robust... That they are disclosing the tools rather than e.g. creating a network service makes this even easier.

jappgar 9mo ago
And now you know the only reason these labs get any funding. It's all to benefit industry, whether the academics realize it or not.

tgv 9mo ago

Idk. Perhaps this technique doesn't work, but if someone comes up with a working system, and LLMs start using techniques to counter it, artists might have a leg to stand upon, as the use of the counter-technique makes clear that the scraper never had any intention of respecting terms of use.

vidarh 9mo ago
They won't need to use counter techniques beyond fixing incorrect output from their models by making the general training methods more robust to features not seen by humans.
pixl97 9mo ago
No, not really. In fact I would say the opposite is true. LLMs must protect against this as a security measure in unified models or things the LLM 'sees' may be faked. If for example someone could trick you into seeing a $1 bill as a $10 it would be considered a huge failure on your part and it would be trained out of you if you wanted to remain employed.

YeGoblynQueenne 9mo ago

>> We’ve seen this arms race before and know who wins. It’s all snake oil imo I haven't and I don't know who wins. Who wins? Adversarial examples aren't snake oil, if that's what you meant. There's a rich literature on both producing and bypassing them that has accumulated over the years, but while I haven't kept abreast with it, my recollection is that the bottom line is like that for online security: there's never a good reason not to make sure your system is up to date and protected from attacks, even if there exist attacks that can bypass any defense. Where in this case attack and defense can both describe what artists want to do with their work.

jappgar 9mo ago
In an arms race, the party with the most money always wins.
- gspr 9mo ago
  Citation needed.
  - deleted 9mo ago
    [deleted]
torginus 9mo ago
Aren't adversarial examples have to be trained to be effective against a specific recognizer? I could imagine you could make one that was effective against multiple recognizers, but not in general. I'd also guess it'd be easy to get rid of this vulnerability on the model side.
pixl97 9mo ago
This isn’t security... Don't confuse attempting to make AI misclassify an image as a security measure. And yes, this is snake oil and the AI wins every time. At the end of the day a human has to be able to interpret the image, and I'd add another constraint of not thinking it looks ugly. This puts a very hard floor on what a poisoner can put in an image before the human gets sick. In a rapid turn around GAN you hit that noise floor really quickly.

oth001 9mo ago

Doesn't mean artists should make it easy for these AI companies to steal artist IP. It doesn't take long to do and seems effective enough from what I've seen. BTW This is how cybersecurity works (cat and mouse etc)

vidarh 9mo ago
The problem is that it is an inherently intractable problem with the (temporary) solution space shrinking with each mitigation, as the images still needs to look good to people.
- pixl97 9mo ago
  Exactly. This isn't like encryption where you can just keep adding more bits. Every iteration that gets closer to simulating how people see sets the floor.
jappgar 9mo ago
Real security systems don't publicize how they work. This is just grandstanding. Half the people from this lab will go on to work for AI companies.
- daeken 9mo ago
  > Real security systems don't publicize how they work. 175 years of history would disagree with you: https://en.wikipedia.org/wiki/Security_through_obscurity https://en.wikipedia.org/wiki/Security_through_obscurity
  - jappgar 9mo ago
    That old saw. Downvote all you want. Adversarial engineering does indeed rely on obscurity, they just don't tell you that.
    - daeken 9mo ago
      I've been working in security for more than 20 years and have seen the deleterious effects of security through obscurity first-hand. Why does "adversarial engineering" rely on obscurity?

zelphirkalt 9mo ago

Isn't there a huge cost imbalance? As in easy to add some noise, difficult to remove reliably, so that even if it gets removed, it could still be counted as a partial win defending against unwanted AI scraping.

cmxch 9mo ago

AI model makers win, luddites lose. Never mind that the more people try to corrupt a model, the more likely that future models will catch these corruption attempts as security and trust/safety issues to fix and work around. The next Nightshade will eventually be viewed as malware to a model and then worked around, reconstructing around the attempt to break a model.