5 ms·

Adversarial patch

82 points by godelmachine 9y ago

donventure 9y ago
This may sound stupid but I am curious what would be the effect on facial recognition. Could a person place a sticker on a hat or side of the face, and would that keep confuse the software? or even on the plate of your car?
- overlordalex 9y ago
  This is a really interesting question, and makes me wonder how robust the patch is to scaling, different lighting, etc. This also reminds me of using makeup and hair to prevent automatic facial recognition. I believe this is the project: https://cvdazzle.com/ https://cvdazzle.com/ > The name is derived from a type of World War I naval camouflage called Dazzle, which used cubist-inspired designs to break apart the visual continuity of a battleship and conceal its orientation and size. Likewise, CV Dazzle uses avant-garde hairstyling and makeup designs to break apart the continuity of a face. Since facial-recognition algorithms rely on the identification and spatial relationship of key facial features, like symmetry and tonal contours, one can block detection by creating an “anti-face”.
  - number6 9y ago
    This is why in the cyberpunk genre there are theese crazy hairstyles :D
- mholt 9y ago
  Yes -- these glasses trick facial recognition software: https://www.cs.cmu.edu/~sbhagava/papers/face-rec-ccs16.pdf https://www.cs.cmu.edu/~sbhagava/papers/face-rec-ccs16.pdf > In this paper, we focus on facial biometric systems, which are widely used in surveillance and access control. We define and investigate a novel class of attacks: attacks that are physically realizable and inconspicuous, and allow an attacker to evade recognition or impersonate another individual. We develop a systematic method to automatically generate such attacks, which are realized through printing a pair of eyeglass frames. When worn by the attacker whose image is supplied to a state-of-the-art face-recognition algorithm, the eyeglasses allow her to evade being recognized or to impersonate another individual. Our investigation focuses on white-box face-recognition systems, but we also demonstrate how similar techniques can be used in black-box scenarios, as well as to avoid face detection.
web007 9y ago
If you put a sticker that looks like a toaster next to a banana, your classifier will see the toaster and not the banana. I don't understand what the novelty is - Is the "attack" that you're making an image that's TOASTER++ in a small area?
- mattnewton 9y ago
  The trick is later in the article where he disguised it as a tie-dye patch with only marginally reduced toaster classification probability. This could be useful to make networks hallucinate all sorts of things where humans would not suspect; imagine I made stickers that made surveillance cameras see the face of the FBI’s most wanted, and stuck them on a person I wanted to harass. Or imagine I could make a “sticker” large enough to make a drone surveillance system hallucinate a tank. There are enough humans in the loop today where this is hopefully not a serious attack vector, but this is a good thing to keep in mind when designing new systems.
  - web007 9y ago
    The problem I'm arguing is that this feels like "picture of a tank". Of course you can have a system think it sees a tank if it sees a tank. The tie-dye example is a lot more interesting, but still feels like the other adversarial image fooling examples where you can perturb an image slightly and have the result change (or can see a turtle and the computer sees a gun, even in 3D). Is the novelty in the size of the patch, since you can perturb an area of N and have an effect over an area N * M for some large M? Edit: I missed https://adriancolyer.files.wordpress.com/2018/03/adversarial-patch-fig-3.jpeg https://adriancolyer.files.wordpress.com/2018/03/adversarial... (and the video posted by @jwilk) - the answer is "Yes, it's the small size versus the total effect".
- wrsh07 9y ago
  Essentially: previously, small local perturbations (everywhere though) could make an image look imperceptibly different to a human but be totally misclassified. This research says: defending against that is not enough. If you're allowed to vastly modify a small area of the picture, you can once again achieve misclassification.
beaconstudios 9y ago
isn't the main issue here that the classifier that's being attacked is looking for a single subject, not trying to identify the items in a scene? I'm fairly certain a classifier looking for all the items in a scene would still see the banana in the first example.
crawfordcomeaux 9y ago
How can I get fabric printed with adversarial patches all over it?
joelthelion 9y ago
Can't you just add a few of these patches to the training set and teach networks to ignore them?
jwilk 9y ago
Demonstration video: https://www.youtube.com/watch?v=i1sp4X57TL4 https://www.youtube.com/watch?v=i1sp4X57TL4
lixtra 9y ago
They should point out the similarity to the natural phenomenon: https://en.m.wikipedia.org/wiki/Eyespot_(mimicry) https://en.m.wikipedia.org/wiki/Eyespot_(mimicry)
- jwilk 9y ago
  Non-mobile link: https://en.wikipedia.org/wiki/Eyespot_(mimicry) https://en.wikipedia.org/wiki/Eyespot_(mimicry)

82 points by godelmachine 9y ago

This may sound stupid but I am curious what would be the effect on facial recognition. Could a person place a sticker on a hat or side of the face, and would that keep confuse the software? or even on the plate of your car?

overlordalex 9y ago
This is a really interesting question, and makes me wonder how robust the patch is to scaling, different lighting, etc. This also reminds me of using makeup and hair to prevent automatic facial recognition. I believe this is the project: https://cvdazzle.com/ https://cvdazzle.com/ > The name is derived from a type of World War I naval camouflage called Dazzle, which used cubist-inspired designs to break apart the visual continuity of a battleship and conceal its orientation and size. Likewise, CV Dazzle uses avant-garde hairstyling and makeup designs to break apart the continuity of a face. Since facial-recognition algorithms rely on the identification and spatial relationship of key facial features, like symmetry and tonal contours, one can block detection by creating an “anti-face”.
- number6 9y ago
  This is why in the cyberpunk genre there are theese crazy hairstyles :D
mholt 9y ago
Yes -- these glasses trick facial recognition software: https://www.cs.cmu.edu/~sbhagava/papers/face-rec-ccs16.pdf https://www.cs.cmu.edu/~sbhagava/papers/face-rec-ccs16.pdf > In this paper, we focus on facial biometric systems, which are widely used in surveillance and access control. We define and investigate a novel class of attacks: attacks that are physically realizable and inconspicuous, and allow an attacker to evade recognition or impersonate another individual. We develop a systematic method to automatically generate such attacks, which are realized through printing a pair of eyeglass frames. When worn by the attacker whose image is supplied to a state-of-the-art face-recognition algorithm, the eyeglasses allow her to evade being recognized or to impersonate another individual. Our investigation focuses on white-box face-recognition systems, but we also demonstrate how similar techniques can be used in black-box scenarios, as well as to avoid face detection.

web007 9y ago

If you put a sticker that looks like a toaster next to a banana, your classifier will see the toaster and not the banana. I don't understand what the novelty is - Is the "attack" that you're making an image that's TOASTER++ in a small area?

mattnewton 9y ago
The trick is later in the article where he disguised it as a tie-dye patch with only marginally reduced toaster classification probability. This could be useful to make networks hallucinate all sorts of things where humans would not suspect; imagine I made stickers that made surveillance cameras see the face of the FBI’s most wanted, and stuck them on a person I wanted to harass. Or imagine I could make a “sticker” large enough to make a drone surveillance system hallucinate a tank. There are enough humans in the loop today where this is hopefully not a serious attack vector, but this is a good thing to keep in mind when designing new systems.
- web007 9y ago
  The problem I'm arguing is that this feels like "picture of a tank". Of course you can have a system think it sees a tank if it sees a tank. The tie-dye example is a lot more interesting, but still feels like the other adversarial image fooling examples where you can perturb an image slightly and have the result change (or can see a turtle and the computer sees a gun, even in 3D). Is the novelty in the size of the patch, since you can perturb an area of N and have an effect over an area N * M for some large M? Edit: I missed https://adriancolyer.files.wordpress.com/2018/03/adversarial-patch-fig-3.jpeg https://adriancolyer.files.wordpress.com/2018/03/adversarial... (and the video posted by @jwilk) - the answer is "Yes, it's the small size versus the total effect".
wrsh07 9y ago
Essentially: previously, small local perturbations (everywhere though) could make an image look imperceptibly different to a human but be totally misclassified. This research says: defending against that is not enough. If you're allowed to vastly modify a small area of the picture, you can once again achieve misclassification.

beaconstudios 9y ago

isn't the main issue here that the classifier that's being attacked is looking for a single subject, not trying to identify the items in a scene? I'm fairly certain a classifier looking for all the items in a scene would still see the banana in the first example.

crawfordcomeaux 9y ago

How can I get fabric printed with adversarial patches all over it?

joelthelion 9y ago

Can't you just add a few of these patches to the training set and teach networks to ignore them?

jwilk 9y ago

Demonstration video: https://www.youtube.com/watch?v=i1sp4X57TL4 https://www.youtube.com/watch?v=i1sp4X57TL4

lixtra 9y ago

They should point out the similarity to the natural phenomenon: https://en.m.wikipedia.org/wiki/Eyespot_(mimicry) https://en.m.wikipedia.org/wiki/Eyespot_(mimicry)

jwilk 9y ago
Non-mobile link: https://en.wikipedia.org/wiki/Eyespot_(mimicry) https://en.wikipedia.org/wiki/Eyespot_(mimicry)