20 ms·

Hacking the Brain with Adversarial Images

97 points by imartin2k 9y ago

deleted 9y ago
[deleted]
matte_black 9y ago
With attacks like this it would never be safe to use image recognition in any applications where human life depends on accurate results.
- dgacmu 9y ago
  Because my favorite hobby is modifying my liver so that the biopsy picture confuses the cancer detection algorithm. :)
  - matte_black 9y ago
    I’m thinking more along the lines of cars, surveillance, heavy construction equipment etc.
    - Filligree 9y ago
      https://xkcd.com/1958/ https://xkcd.com/1958/
      - matte_black 9y ago
        If this is the defense, it’s paper thin. Human drivers can easily reason about fake lanes, or fake pedestrian cut outs, or a sign somewhere that it shouldn’t be. You also don’t need an abundance of murderers.
  - mikeash 9y ago
    You modify someone else’s liver, as a quick and easy method of assassination.
lambada 9y ago
This reminds me strongly of the BLIT series of short stories, in which a category of images is discovered that cause processing problems ‘glitches’ in the human brain with fatal results. http://www.infinityplus.co.uk/stories/blit.htm http://www.infinityplus.co.uk/stories/blit.htm That’s the first one, others are available online too.
- taneq 9y ago
  The other work of popular fiction that gets referenced in these cases is, of course, Snow Crash.
yters 9y ago
So if I make a cat look like a dog, people think it's a dog. That's amazing and totally unexpected.
- saulrh 9y ago
  The point is that the human brain is just as vulnerable to the adversarial attacks that people are claiming will be the downfall of ML. It's like the people that scream about people shining lasers into the cameras of autonomous vehicles - yeah, sure, if you did the same thing to a human driver you'd get exactly the same crash, why are you suddenly worried about this? And it's more than just "making it look like a dog". If you look at it for any length of time it's obvious that it's a picture of a cat with some noise on it. But if you only have 100ms to respond to it you are guaranteed (>95% misidentification) to say that it's a cat. Now what happens if you take this and make your car look like a tree? Someone could crash and that would be terrible! Cancel driving, tear up the roads, ban wheels, cars were a mistake! /s
  - matte_black 9y ago
    It’s not the same. Shining lasers into eyes is not scalable. However, placing a little sign at the side of a road that causes self driving vehicles to swerve out and crash violently, killing the occupants, can be done in mass.
    - saulrh 9y ago
      If you make the sign red and octagonal you could even kill human drivers too! Or maybe if you slapped an illusory-motion pattern over something that looked like a dog, then people might swerve into the opposite lane to avoid it, double kill! Shock, horror! Human vision is slightly more robust solely because it's had more time to go back and forth with adversaries. Nothing prevents ML from reaching the same levels of safety. Nothing prevents you from deploying attacks against humans that're identical to to the attacks against artificial systems.
    - thisacctforreal 9y ago
      Further, laser beaming eyes isn't an intellectual pursuit; entering an arms race with some of the world's top engineers certainly is. I imagine most self-driving car hackers will react to any successes with "holy shit it worked" followed by remorse.
trevyn 9y ago
Article subheading: >Researchers from Google Brain show that adversarial images can trick both humans and computers, and the implications are scary So the IEEE is telling me how to feel about this article in addition to presenting the facts. One might even consider that "hacking the brain with adversarial text". :-) >A worrying possibility is that supernormal stimuli designed to influence human behavior or emotions... Sooooo, like clickbait headlines? :-)
- TheOtherHobbes 9y ago
  Exactly like clickbait headlines, political posturing, advertising, and propaganda - all of which are misleading stimuli designed to elicit a specific behavioural and cognitive response. Brains are very easy to hack. The idea that we're reliable exemplars of rational objectivity and rigorous self-awareness is nonsense.
deleted 9y ago
[deleted]
arkades 9y ago
The subsequent images and conclusion that adverserial attacks against multiple models would be effective against humans seems tautological. Individual models may be unique, but multiple models are related in that -human- classifiers are the final source of truth. Anything that works against all of them is indirectly targeting the human classifiers that unite them.
kerng 9y ago
This is interesting. Could this be a way to watermark images automatically before uploading to the "cloud" to prevent AI processing by large corporations (e.g. Facebook)? Main problem seems that the AI can be adjusted down the road to eventually fix it.
lowracle 9y ago
How far fetched would it be to assume that in the future, facebook would start building a neural network for each of its users, training it to like the same content as you ? With the amount of data generated by the endless scrolling, you could build a fairly accurate neural network of which content produce which emotion, and then generate adversarial images for targeted advertising.
- AnIdiotOnTheNet 9y ago
  At some point that neural network could just take over your social media presence and no one would ever know, because it's essentially your clone.
ccvannorman 9y ago
TLDR: If you modify pixels enough, you confuse humans, especially when you only let the humans look at the tiny blurry image for 70 ms. rolls eyes I feel like this article totally misses the point about why adversarial ML tactics are interesting. The fundamental reason they work is that computers don't have any abstract notion of a cat, a toaster, a banana, or even gravity or matter. THAT is why they're easily fooled. To stack up two adversarial images and say, "Look we fooled humans too, OMG humans are just like computers!" is like saying, "We erased the lines on the road and both self-driving cars AND humans got confused, OMG humans == computers!" I know the article isn't saying humans == computers, but .. come on .. I'm not seeing any merit to this investigation
YeGoblynQueenne 9y ago
Woooah. Huuge assumptions here. First of all, when we say that a classifier "misclassifies" an image, we have an agreed-upon definition of what that means: it means we have an image, we have a target label for it and the classifier assigns it a different label. What exactly does it mean that a human misclassifies an image? More specifically, when the article says that "humans think that they're looking at something they aren't", what in the world does that mean? I mean, I have to assume a confusion between "looking", and "seeing". I can't really believe the article is saying that when I'm looking at the top right image I'm not actually, you know looking at it; but instead ...looking outside the window? Or what? On the other hand- seeing? Really? Who knows what it is that anyone else is seeing? Who knows what is there to see? Especially when it comes to an image specifically manipulated to be confusing, as opposed to a real-world image? What is the ground truth here, for the image on the top-right? Is it a cat, just because Google reserachers say it's a cat and it's fooling my senses into thinking it's a dog ish? Is it fooling my senses just because the Google researchers say it is? Google researchers are the final arbiters of objective visible reality, now? To be more precise, who is to say that that image can just be a "cat" or a "dog" and nothineg else? Because, you know, the first thing I thought when I saw that image on the top right was "that looks like a jumbled mess made with bits of an image of a cat and an image of a dog", or something along those lines. Of course, if you sit me in front of a computer with two buttons, one for "dog" and one for "cat" like in the researchers' setup... well, then you can force me to misclassify the image all you like. But what does that prove? Besides the fact that if you force me to choose one of two things, without knowing what you think is the "right" thing to choose, I'll choose the "wrong" thing a lot of the time?
debatem1 9y ago
Seems like an odd approach. The interesting thing about robust adversarial examples is that they fool computers but not humans. Something that fooled humans but not computers would seem to be the right next step. If you fool both it seems more likely that you just made an image that looks like what it classifies as.
- stanfordkid 9y ago
  Yeah my first thought was - well dogs and cats look kind of similar under fairly simple transformations. What would be interesting is if you could establish a correspondence between visual and genetic similarity... neural networks provide a distance metric.
ouid 9y ago
there's a lot of bullshit in this article. To start, it should be clear to everyone that it is possible to actually transform a picture of a cat into a picture of a dog. The argument that the author is trying to make is not, in fact, that the human brain is hackable with adversarial images, but that the kinds of strategies which allow you to hack a wide range of deep learning models (to turn cats into dogs) require the introduction of "features" that a human would recognize as being doglike. The leading image is not evidence of this, however. We have to keep track of the what figures correspond to which claims, because the article jumps around. The first figure shows us a mask which people agree turns a cat into a dog. This is not evidence of any claim because such masks are guaranteed to exist. The second figure shows a mask which people don't even detect which causes a computer to claim that a panda is a gibbon. This is basically the null hypothesis of the paper, it demonstrates that you can attack a single computer with a subtle mask. The third figure shows us an example of an attack that is robust across deep learning models and also to perspective shifts. The argument being defended by this figure is that humans recognize the mistakes that the machine is making as being non-superficial features of laptops and toasters. The author undermines his own argument here, by requiring that the attack be robust to perspective shifts. The point he/she is trying to make is that in order to be robust to many neural nets you have to include non-superficial features, but then convolutes that claim by also requiring the features to be robust to other types of transformation as well. Does the "featuriness" of the attack come from the multiple neural network requirement or for the transformational robustness requirement? The answer is unknown. Furthermore, I think this argument falls short pretty quickly if you simply hide the computer's answer. Would you guess that the computer guessed laptop for the first image, even if you knew it got it wrong? How far down your list would that have been? Or toaster for the second? The features offered in the second image definitely look like they have some features that are similar to the corner of a rounded metal cube, but this is exactly a superficial image statistic which is robust to change of perspective transformations. My classifier might as well be looking for that exactly. This is not evidence for the authors case. The fourth and fifth figures are more interesting, but still do not provide any evidence in defense of machine learning. It's important, for some reason, to point out that the accuracy on image tasks is NOT measured linearly with error rate. 99.9% accuracy is 10 times better than 99% accuracy. The difference between 65 and 75% is about 1000 times less than the difference between 75% and 99% Allow me to submit, without evidence, that humans can differentiate between a cat and a dog with >99% accuracy when given as much time as they need to make a decision about whether or not they are looking at a cat or a dog (with no adversarial mask applied). This means that the difference between human performance on the task without the time constraint, versus human performance on the task with the time constraint, is about 1000 times larger than the difference between human performance on the task with the time constraint and the task with the time constraint and the filter. This means that if you really want people to fail on the cats versus dogs task, spending time coming up with the filter to apply after you've somehow managed to present the image that you're trying to get the person to misclassify for 1/20 of a second, followed by a noise mask, is >100 times less valuable at getting them to misclassify the picture than figuring out how to add the time constraint. Not to mention the fact that transformations that actually take cats to dogs and vice versa are necessarily possible, and that differentiation between cats and dogs is an image recognition task with a much clearer limit on human performance than differentiation between cats and laptops, which we are nevertheless amazingly good at. Cats are very close to dogs, and it seems like this paper has completely forgotten this fact, and is deeply surprised to learn it. Overall this is a very poor defense of deep learning.
m3kw9 9y ago
How is this news? It’s wraping AI terminology over what we would call over since dawn of human, a disguise.

97 points by imartin2k 9y ago

deleted 9y ago

[deleted]

With attacks like this it would never be safe to use image recognition in any applications where human life depends on accurate results.

dgacmu 9y ago
Because my favorite hobby is modifying my liver so that the biopsy picture confuses the cancer detection algorithm. :)
- matte_black 9y ago
  I’m thinking more along the lines of cars, surveillance, heavy construction equipment etc.
  - Filligree 9y ago
    https://xkcd.com/1958/ https://xkcd.com/1958/
    - matte_black 9y ago
      If this is the defense, it’s paper thin. Human drivers can easily reason about fake lanes, or fake pedestrian cut outs, or a sign somewhere that it shouldn’t be. You also don’t need an abundance of murderers.
- mikeash 9y ago
  You modify someone else’s liver, as a quick and easy method of assassination.

lambada 9y ago

This reminds me strongly of the BLIT series of short stories, in which a category of images is discovered that cause processing problems ‘glitches’ in the human brain with fatal results. http://www.infinityplus.co.uk/stories/blit.htm http://www.infinityplus.co.uk/stories/blit.htm That’s the first one, others are available online too.

taneq 9y ago
The other work of popular fiction that gets referenced in these cases is, of course, Snow Crash.

yters 9y ago

So if I make a cat look like a dog, people think it's a dog. That's amazing and totally unexpected.

saulrh 9y ago
The point is that the human brain is just as vulnerable to the adversarial attacks that people are claiming will be the downfall of ML. It's like the people that scream about people shining lasers into the cameras of autonomous vehicles - yeah, sure, if you did the same thing to a human driver you'd get exactly the same crash, why are you suddenly worried about this? And it's more than just "making it look like a dog". If you look at it for any length of time it's obvious that it's a picture of a cat with some noise on it. But if you only have 100ms to respond to it you are guaranteed (>95% misidentification) to say that it's a cat. Now what happens if you take this and make your car look like a tree? Someone could crash and that would be terrible! Cancel driving, tear up the roads, ban wheels, cars were a mistake! /s
- matte_black 9y ago
  It’s not the same. Shining lasers into eyes is not scalable. However, placing a little sign at the side of a road that causes self driving vehicles to swerve out and crash violently, killing the occupants, can be done in mass.
  - saulrh 9y ago
    If you make the sign red and octagonal you could even kill human drivers too! Or maybe if you slapped an illusory-motion pattern over something that looked like a dog, then people might swerve into the opposite lane to avoid it, double kill! Shock, horror! Human vision is slightly more robust solely because it's had more time to go back and forth with adversaries. Nothing prevents ML from reaching the same levels of safety. Nothing prevents you from deploying attacks against humans that're identical to to the attacks against artificial systems.
  - thisacctforreal 9y ago
    Further, laser beaming eyes isn't an intellectual pursuit; entering an arms race with some of the world's top engineers certainly is. I imagine most self-driving car hackers will react to any successes with "holy shit it worked" followed by remorse.

trevyn 9y ago

Article subheading: >Researchers from Google Brain show that adversarial images can trick both humans and computers, and the implications are scary So the IEEE is telling me how to feel about this article in addition to presenting the facts. One might even consider that "hacking the brain with adversarial text". :-) >A worrying possibility is that supernormal stimuli designed to influence human behavior or emotions... Sooooo, like clickbait headlines? :-)

TheOtherHobbes 9y ago
Exactly like clickbait headlines, political posturing, advertising, and propaganda - all of which are misleading stimuli designed to elicit a specific behavioural and cognitive response. Brains are very easy to hack. The idea that we're reliable exemplars of rational objectivity and rigorous self-awareness is nonsense.

deleted 9y ago

[deleted]

arkades 9y ago

The subsequent images and conclusion that adverserial attacks against multiple models would be effective against humans seems tautological. Individual models may be unique, but multiple models are related in that -human- classifiers are the final source of truth. Anything that works against all of them is indirectly targeting the human classifiers that unite them.

kerng 9y ago

This is interesting. Could this be a way to watermark images automatically before uploading to the "cloud" to prevent AI processing by large corporations (e.g. Facebook)? Main problem seems that the AI can be adjusted down the road to eventually fix it.

lowracle 9y ago

How far fetched would it be to assume that in the future, facebook would start building a neural network for each of its users, training it to like the same content as you ? With the amount of data generated by the endless scrolling, you could build a fairly accurate neural network of which content produce which emotion, and then generate adversarial images for targeted advertising.

AnIdiotOnTheNet 9y ago
At some point that neural network could just take over your social media presence and no one would ever know, because it's essentially your clone.

ccvannorman 9y ago

TLDR: If you modify pixels enough, you confuse humans, especially when you only let the humans look at the tiny blurry image for 70 ms. rolls eyes I feel like this article totally misses the point about why adversarial ML tactics are interesting. The fundamental reason they work is that computers don't have any abstract notion of a cat, a toaster, a banana, or even gravity or matter. THAT is why they're easily fooled. To stack up two adversarial images and say, "Look we fooled humans too, OMG humans are just like computers!" is like saying, "We erased the lines on the road and both self-driving cars AND humans got confused, OMG humans == computers!" I know the article isn't saying humans == computers, but .. come on .. I'm not seeing any merit to this investigation

YeGoblynQueenne 9y ago

Woooah. Huuge assumptions here. First of all, when we say that a classifier "misclassifies" an image, we have an agreed-upon definition of what that means: it means we have an image, we have a target label for it and the classifier assigns it a different label. What exactly does it mean that a human misclassifies an image? More specifically, when the article says that "humans think that they're looking at something they aren't", what in the world does that mean? I mean, I have to assume a confusion between "looking", and "seeing". I can't really believe the article is saying that when I'm looking at the top right image I'm not actually, you know looking at it; but instead ...looking outside the window? Or what? On the other hand- seeing? Really? Who knows what it is that anyone else is seeing? Who knows what is there to see? Especially when it comes to an image specifically manipulated to be confusing, as opposed to a real-world image? What is the ground truth here, for the image on the top-right? Is it a cat, just because Google reserachers say it's a cat and it's fooling my senses into thinking it's a dog ish? Is it fooling my senses just because the Google researchers say it is? Google researchers are the final arbiters of objective visible reality, now? To be more precise, who is to say that that image can just be a "cat" or a "dog" and nothineg else? Because, you know, the first thing I thought when I saw that image on the top right was "that looks like a jumbled mess made with bits of an image of a cat and an image of a dog", or something along those lines. Of course, if you sit me in front of a computer with two buttons, one for "dog" and one for "cat" like in the researchers' setup... well, then you can force me to misclassify the image all you like. But what does that prove? Besides the fact that if you force me to choose one of two things, without knowing what you think is the "right" thing to choose, I'll choose the "wrong" thing a lot of the time?

debatem1 9y ago

Seems like an odd approach. The interesting thing about robust adversarial examples is that they fool computers but not humans. Something that fooled humans but not computers would seem to be the right next step. If you fool both it seems more likely that you just made an image that looks like what it classifies as.

stanfordkid 9y ago
Yeah my first thought was - well dogs and cats look kind of similar under fairly simple transformations. What would be interesting is if you could establish a correspondence between visual and genetic similarity... neural networks provide a distance metric.

ouid 9y ago

there's a lot of bullshit in this article. To start, it should be clear to everyone that it is possible to actually transform a picture of a cat into a picture of a dog. The argument that the author is trying to make is not, in fact, that the human brain is hackable with adversarial images, but that the kinds of strategies which allow you to hack a wide range of deep learning models (to turn cats into dogs) require the introduction of "features" that a human would recognize as being doglike. The leading image is not evidence of this, however. We have to keep track of the what figures correspond to which claims, because the article jumps around. The first figure shows us a mask which people agree turns a cat into a dog. This is not evidence of any claim because such masks are guaranteed to exist. The second figure shows a mask which people don't even detect which causes a computer to claim that a panda is a gibbon. This is basically the null hypothesis of the paper, it demonstrates that you can attack a single computer with a subtle mask. The third figure shows us an example of an attack that is robust across deep learning models and also to perspective shifts. The argument being defended by this figure is that humans recognize the mistakes that the machine is making as being non-superficial features of laptops and toasters. The author undermines his own argument here, by requiring that the attack be robust to perspective shifts. The point he/she is trying to make is that in order to be robust to many neural nets you have to include non-superficial features, but then convolutes that claim by also requiring the features to be robust to other types of transformation as well. Does the "featuriness" of the attack come from the multiple neural network requirement or for the transformational robustness requirement? The answer is unknown. Furthermore, I think this argument falls short pretty quickly if you simply hide the computer's answer. Would you guess that the computer guessed laptop for the first image, even if you knew it got it wrong? How far down your list would that have been? Or toaster for the second? The features offered in the second image definitely look like they have some features that are similar to the corner of a rounded metal cube, but this is exactly a superficial image statistic which is robust to change of perspective transformations. My classifier might as well be looking for that exactly. This is not evidence for the authors case. The fourth and fifth figures are more interesting, but still do not provide any evidence in defense of machine learning. It's important, for some reason, to point out that the accuracy on image tasks is NOT measured linearly with error rate. 99.9% accuracy is 10 times better than 99% accuracy. The difference between 65 and 75% is about 1000 times less than the difference between 75% and 99% Allow me to submit, without evidence, that humans can differentiate between a cat and a dog with >99% accuracy when given as much time as they need to make a decision about whether or not they are looking at a cat or a dog (with no adversarial mask applied). This means that the difference between human performance on the task without the time constraint, versus human performance on the task with the time constraint, is about 1000 times larger than the difference between human performance on the task with the time constraint and the task with the time constraint and the filter. This means that if you really want people to fail on the cats versus dogs task, spending time coming up with the filter to apply after you've somehow managed to present the image that you're trying to get the person to misclassify for 1/20 of a second, followed by a noise mask, is >100 times less valuable at getting them to misclassify the picture than figuring out how to add the time constraint. Not to mention the fact that transformations that actually take cats to dogs and vice versa are necessarily possible, and that differentiation between cats and dogs is an image recognition task with a much clearer limit on human performance than differentiation between cats and laptops, which we are nevertheless amazingly good at. Cats are very close to dogs, and it seems like this paper has completely forgotten this fact, and is deeply surprised to learn it. Overall this is a very poor defense of deep learning.

m3kw9 9y ago

How is this news? It’s wraping AI terminology over what we would call over since dawn of human, a disguise.