5 ms·

If you put a sticker that looks like a toaster next to a banana, your classifier will see the toaster and not the banana. I don't understand what the novelty i

by web007 9y ago

If you put a sticker that looks like a toaster next to a banana, your classifier will see the toaster and not the banana. I don't understand what the novelty is - Is the "attack" that you're making an image that's TOASTER++ in a small area?

mattnewton 9y ago
The trick is later in the article where he disguised it as a tie-dye patch with only marginally reduced toaster classification probability. This could be useful to make networks hallucinate all sorts of things where humans would not suspect; imagine I made stickers that made surveillance cameras see the face of the FBI’s most wanted, and stuck them on a person I wanted to harass. Or imagine I could make a “sticker” large enough to make a drone surveillance system hallucinate a tank. There are enough humans in the loop today where this is hopefully not a serious attack vector, but this is a good thing to keep in mind when designing new systems.
- web007 9y ago
  The problem I'm arguing is that this feels like "picture of a tank". Of course you can have a system think it sees a tank if it sees a tank. The tie-dye example is a lot more interesting, but still feels like the other adversarial image fooling examples where you can perturb an image slightly and have the result change (or can see a turtle and the computer sees a gun, even in 3D). Is the novelty in the size of the patch, since you can perturb an area of N and have an effect over an area N * M for some large M? Edit: I missed https://adriancolyer.files.wordpress.com/2018/03/adversarial-patch-fig-3.jpeg https://adriancolyer.files.wordpress.com/2018/03/adversarial... (and the video posted by @jwilk) - the answer is "Yes, it's the small size versus the total effect".
wrsh07 9y ago
Essentially: previously, small local perturbations (everywhere though) could make an image look imperceptibly different to a human but be totally misclassified. This research says: defending against that is not enough. If you're allowed to vastly modify a small area of the picture, you can once again achieve misclassification.
- wrsh07 9y ago
  Also: > This departs from most prior work on adversarial perturbations in the fact that this perturbation is universal in that it works for any background.
Spivak 9y ago
Attacks against computer vision algorithms are pretty much always going to be of the form of "I can make a change to the scene or image which makes no difference to a human but which causes an erroneous classification." This is a huge problem if we expect autonomous systems to do anything meaningful with vision outside of tightly controlled environments. You're going to be crucified after an autonomous vehicle merrily plows into a semi and the explanation for the crash is "well the truck was white and had the Hidden Valley logo on it which was misinterpreted as clouds." Or if an autonomous car runs a red because there was billboard somewhere behind the light which sorta-kinda looks like a green circle.
- psychometry 9y ago
  Or meddling kids put stickers on stop signs that make them look like no parking signs to an Uber automated car.
- wrsh07 9y ago
  Semi relevant xkcd: https://xkcd.com/1958/ https://xkcd.com/1958/ But these attacks are worth thinking about, and it is definitely useful to build systems that are robust to this type of meddling.
candiodari 9y ago
> If you put a sticker that looks like a toaster next to a banana, your classifier will see the toaster and not the banana. No it won't (as demonstrated in the video). This is the direct result of having CNNs. CNNs recognize pictures because of what a (at most) 32x32 set of pixels look like. It recognizes 1024 pixel patches as being toaster or not. So no, a toaster for an image classifier is VERY different from a toaster. It's how the corners of the surface work. It's the grid within the top. It's the reflecting aluminum near the plastic border. It's the ... A sticker doesn't have the vast majority of those properties. So a sticker with a toaster doesn't actually (usually) cause a misclassification (one imagines you can of course make the sticker good enough, but ...). But you can collect a lot of those 32x32 patches that are really, really "toastery" and then scale them, find something that is "somewhat fractal" (looks the same at different scales) and put that on a sticker. That looks nothing at all like a toaster to you, but it looks a hell of a lot like a toaster to a CNN. If you learn CNNs by knowing their predecessor "trick" (google "haar cascade") this makes a lot more sense. The reason we aren't fooled* is that our eyes don't recognize pixels. We recognize a wavelet encoding of a picture, not a grid of pixels. We recognize something that's like this description "a red blob, with a smaller green blob in the 2 o'clock position, a blue blob near the bottom, then that green blob is actually split into light green on the left and dark green on the right, and that blue blob really has a white top and a black bottom and ..." (you see you give details about a large average then afterwards describe the parts where that average is wrong. Rather than describing a grid top-left to bottom right you describe it as "average of the whole grid X, average of top left Y, average of top right B, average of bottom left Z, average of bottom right A, then recurse). * truth is you can fool humans partially with these stickers. You can make stickers that clearly are plastic stickers but you strongly feel that if you tough them you'll feel a cat's fur, or an insect/spider, or ... despite looking nothing like those things, and of course you can tell they're stickers). And of course, there's Dali.

by web007 9y ago

mattnewton 9y ago

The trick is later in the article where he disguised it as a tie-dye patch with only marginally reduced toaster classification probability. This could be useful to make networks hallucinate all sorts of things where humans would not suspect; imagine I made stickers that made surveillance cameras see the face of the FBI’s most wanted, and stuck them on a person I wanted to harass. Or imagine I could make a “sticker” large enough to make a drone surveillance system hallucinate a tank. There are enough humans in the loop today where this is hopefully not a serious attack vector, but this is a good thing to keep in mind when designing new systems.

web007 9y ago
The problem I'm arguing is that this feels like "picture of a tank". Of course you can have a system think it sees a tank if it sees a tank. The tie-dye example is a lot more interesting, but still feels like the other adversarial image fooling examples where you can perturb an image slightly and have the result change (or can see a turtle and the computer sees a gun, even in 3D). Is the novelty in the size of the patch, since you can perturb an area of N and have an effect over an area N * M for some large M? Edit: I missed https://adriancolyer.files.wordpress.com/2018/03/adversarial-patch-fig-3.jpeg https://adriancolyer.files.wordpress.com/2018/03/adversarial... (and the video posted by @jwilk) - the answer is "Yes, it's the small size versus the total effect".

wrsh07 9y ago

Essentially: previously, small local perturbations (everywhere though) could make an image look imperceptibly different to a human but be totally misclassified. This research says: defending against that is not enough. If you're allowed to vastly modify a small area of the picture, you can once again achieve misclassification.

wrsh07 9y ago
Also: > This departs from most prior work on adversarial perturbations in the fact that this perturbation is universal in that it works for any background.

Spivak 9y ago

Attacks against computer vision algorithms are pretty much always going to be of the form of "I can make a change to the scene or image which makes no difference to a human but which causes an erroneous classification." This is a huge problem if we expect autonomous systems to do anything meaningful with vision outside of tightly controlled environments. You're going to be crucified after an autonomous vehicle merrily plows into a semi and the explanation for the crash is "well the truck was white and had the Hidden Valley logo on it which was misinterpreted as clouds." Or if an autonomous car runs a red because there was billboard somewhere behind the light which sorta-kinda looks like a green circle.

psychometry 9y ago
Or meddling kids put stickers on stop signs that make them look like no parking signs to an Uber automated car.
wrsh07 9y ago
Semi relevant xkcd: https://xkcd.com/1958/ https://xkcd.com/1958/ But these attacks are worth thinking about, and it is definitely useful to build systems that are robust to this type of meddling.

candiodari 9y ago

> If you put a sticker that looks like a toaster next to a banana, your classifier will see the toaster and not the banana. No it won't (as demonstrated in the video). This is the direct result of having CNNs. CNNs recognize pictures because of what a (at most) 32x32 set of pixels look like. It recognizes 1024 pixel patches as being toaster or not. So no, a toaster for an image classifier is VERY different from a toaster. It's how the corners of the surface work. It's the grid within the top. It's the reflecting aluminum near the plastic border. It's the ... A sticker doesn't have the vast majority of those properties. So a sticker with a toaster doesn't actually (usually) cause a misclassification (one imagines you can of course make the sticker good enough, but ...). But you can collect a lot of those 32x32 patches that are really, really "toastery" and then scale them, find something that is "somewhat fractal" (looks the same at different scales) and put that on a sticker. That looks nothing at all like a toaster to you, but it looks a hell of a lot like a toaster to a CNN. If you learn CNNs by knowing their predecessor "trick" (google "haar cascade") this makes a lot more sense. The reason we aren't fooled* is that our eyes don't recognize pixels. We recognize a wavelet encoding of a picture, not a grid of pixels. We recognize something that's like this description "a red blob, with a smaller green blob in the 2 o'clock position, a blue blob near the bottom, then that green blob is actually split into light green on the left and dark green on the right, and that blue blob really has a white top and a black bottom and ..." (you see you give details about a large average then afterwards describe the parts where that average is wrong. Rather than describing a grid top-left to bottom right you describe it as "average of the whole grid X, average of top left Y, average of top right B, average of bottom left Z, average of bottom right A, then recurse). * truth is you can fool humans partially with these stickers. You can make stickers that clearly are plastic stickers but you strongly feel that if you tough them you'll feel a cat's fur, or an insect/spider, or ... despite looking nothing like those things, and of course you can tell they're stickers). And of course, there's Dali.