11 ms·
One pixel attack for deceiving deep neural networks
- jaimex2 9y agoInteresting, can this be prevented by simply random blurring images first? The NN should still be able to recognise something partially blurred if trained that way.
- draugadrotten 9y agoor perhaps adding random pixel noise to the image before attempting training/recognition to prevent "smoothness" in an area from being a recognized property
- viraptor 9y agoAlternatively eliminate the noise by low-pass filtering in freq domain. Even a small LED light will bleed to more than one pixel in a real photo.
- mattkrause 9y agoI don't want to totally brush off the security implications, but I think focusing on ways to mitigate this exact "attack" is almost missing the point. You could trivially thwart this attack by rate-limiting--it needs many passes through the network to evolve an image--or by caching the classifier's output and returning it for all similar (e.g., by hamming distance) images. Instead, I think this work is interesting because it shows the limits of the network's generalization abilities.
- taneq 9y agoWhen discussing adversarial attacks on neural nets, everyone seems to just kind of take for granted that the human visual system is immune to this kind of fudgery. I'd be very interested (and a little afraid) to know if this is in fact the case - we know of a bunch of optical illusions that are perceived differently to what they "ought to be", so it's not too much of a stretch to figure that we could come up with a technique to generate more extreme examples.
- sjg007 9y agoPeople do generate optical illusions of the time the red green gold dress is a modern example.
- yodon 9y agoIt’s still conjecture but the chances are good that photosensitive epilepsy[0] as triggered for example by the famous Pokemon episode[1] is conceptually similar to the unstable responses that adversarial attacks produce in computational neural networks (and yes while there is some debate about the total number of seizures triggered by the broadcasting of that episode there is no debate as to whether the medical condition exists and is an real risk factor for a surprisingly high fraction of the population, particularly males and children[2]) [0]https://en.wikipedia.org/wiki/Photosensitive_epilepsy https://en.wikipedia.org/wiki/Photosensitive_epilepsy [1]https://en.wikipedia.org/wiki/Dennō_Senshi_Porygon https://en.wikipedia.org/wiki/Dennō_Senshi_Porygon [2]https://www.epilepsysociety.org.uk/photosensitive-epilepsy https://www.epilepsysociety.org.uk/photosensitive-epilepsy
- hammock 9y ago>It’s still conjecture but the chances are good that photosensitive epilepsy[0] as triggered for example by the famous Pokemon episode[1] is conceptually similar to the unstable responses that adversarial attacks produce in computational neural networks If you believe that then you might also believe there exists a "brown noise" like in Southpark that can cause people's bowels to release..
- jszymborski 9y agoSo, I might be wrong, but my understanding of these attacks is that they require you know what the model of the classifier is. If my understanding is correct, I guess my question is: how general are these attacks? Can we ever say "oh yah, don't try to classify this penguin with your off-the-shelf model, it'll come-up iguana 9/10"
- rlanday 9y agoYou don't necessarily need to know the whole model, but you need accurate enough prediction score outputs for each category. I think if you have that, that's enough to numerically approximate the gradient of the output scores and perform gradient descent.
- zeitbeule 9y agoSomeone just published a new attack at ICLR that does not even need confidence scores but works solely on the final decision output of the network and works as well as the best known attacks (that so far required gradients): https://openreview.net/forum?id=SyZI0GWCZ¬eId=SyZI0GWCZ https://openreview.net/forum?id=SyZI0GWCZ¬eId=SyZI0GWCZ
- Donald 9y agoSee https://arxiv.org/abs/1610.08401 https://arxiv.org/abs/1610.08401 for model agnostic examples of adversarial attacks.
- IshKebab 9y agoThis is much more interesting, thanks!
- jszymborski 9y agoThis is great stuff! I've noticed that the inter-model universality is OK (a little over 50% for one of the universal pertubations), but that's still pretty good!
- justifier 9y ago
- kmbriedis 9y agoDoes this publication bring any value as it was known well before that such thing is possible? Or no-one had measured how many pictures can be "transformed" by changing only 1 pixel?
- bfirsh 9y agoIf you're on a phone, here's a responsive HTML version of the paper: https://www.arxiv-vanity.com/papers/1710.08864/ https://www.arxiv-vanity.com/papers/1710.08864/
- curiousgal 9y agoWoah, thank you!
- jwilk 9y agos/If you're on a phone,// Reading PDFs on screen sucks.
- jwilk 9y agoBeware that figure numbers in the HTML version are incorrect. :(
- metaphor 9y agoFigure 8. Without a label, I wouldn't have been able to independently tell anyone that this was a picture of a dog. For what it's worth, images used in this paper are 32x32 pixels.
- JepZ 9y agoI can't even see a dog when I know there should be one...
- wingerlang 9y agoLooks more like one of these dump trucks to me http://3.bp.blogspot.com/__EzFEHn2YBI/SbEdsBdbL7I/AAAAAAAAARY/wDqn0-t74R0/s1600/cat797.jpg http://3.bp.blogspot.com/__EzFEHn2YBI/SbEdsBdbL7I/AAAAAAAAAR... (why are these always yellow?) Or half a torso of a dog... with a dump truck in the background..
- mattkrause 9y ago> (why are these always yellow? The yellow is the manufacturer's (Caterpillar) trademark color--and it probably helps that it's very high visibility.
- wingerlang 9y agoThat would be my guess, but I see lots of other types of vehicles that are normally yellow in other colors. But this specific kind I don't recall ever seeing in another color. Even here [0] most are yellow despite having a different manufacturer. Maybe the difference is enough to differentiate them though. [0] https://en.wikipedia.org/wiki/Haul_truck#Ultra_class https://en.wikipedia.org/wiki/Haul_truck#Ultra_class
- Animats 9y agoThe underlying problem seems to be that deep neural network classifiers tend to place their classification boundary surfaces very close to data points in at least one dimension in a high-dimensional space. That makes them brittle - perturb the data very slightly in the wrong direction and they move through a boundary into some other classification. I don't know enough about the subject to know why training does that, or what can be done about it.
- mamp 9y agoWhat's concerning is that the network used dropout, which I thought was aimed and making networks less brittle (i.e. reduce overfitting).
- RSchaeffer 9y agoGo watch Yatin Gal's talk on dropout in neural networks. He shows pretty convincingly that the belief that dropout reduces network overfitting by introducing noise is wrong.
- mannigfaltig 9y agoWait, that can’t be wrong because that is literally what DO does. It is a convex hull regularizer around the network activations using noise. That is also why dropout does not solve susceptibility to adversarial examples: It merely extends the regions that the NN generalizes to outward; but that is limited because high-dimensional spaces are counter-intuitively large and the noise required to cover a descent fraction of the “unmapped” space would completely prevent learning. AFAIK, Yarin Gal merely provides a Bayesian interpretation of the noise.
- RSchaeffer 9y agoIIRC, his "Bayesian interpretation of the noise" actually shows that dropout performs approximate integration over model parameters. As he says, dropout doesn't work because of the noise but despite the noise. https://youtu.be/3ONLxYeM1Sc?t=19m21s https://youtu.be/3ONLxYeM1Sc?t=19m21s
- knolan 9y agoWouldn’t it be sensible to median filter the input images to remove these types of single pixel outliers?
- tbabb 9y agoIf by knowing the structure of a NN a "trick" image can be specially crafted to fool it, what is the equivalent for a human brain? If we get to the point of having a human connectome to analyze-- or the kind of access to neural topology that a neural lace would provide-- could an optimizer generate an image of static that human would mistake for the president of the United States? It seems outwardly implausible that such an image could exist, but perhaps that is only because we've never seen one (or if we had, would we know?), and a "blind" search of images would never find it, as the space of images is galactically huge. With a "map" of the brain it might suddenly become plausible. And if so, that world sounds absolutely terrifying to me.
- halflings 9y agoThe human brain is incredibly more complex than a "simple" (artificial) neural network, so there might very well be no equivalent for a real brain... but I think optical illusions would be a good approximation, even if less severe: tricking the brain in elaborate ways to see something that is different from what it would usually see.
- govg 9y agoThe brain has some inherent biases and prejudices - these arise from just how we grow up and perceive objects. So optical illusions are a deviation from whatever visuals we are used to, and you can say similar events occur with stuff like tasting an unknown object, or finding a different smell.
- TeMPOraL 9y agoI'd look at the superset containing both optical illusions and the the various funny brain exploits lkie rerarnagnig ltetres in wrods, or using more than one "the" in a sentence without reader noticing it. I don't know what's the the name of that superset, but I don't think we should limit study of our visual processing stack to object recognition - the brain does so much more!
- Peaker 9y agoI'd expect redundancies in analysis in the brain's NN make this impossible.
- alexott 9y agoRelated article: "BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain" (https://arxiv.org/abs/1708.06733 https://arxiv.org/abs/1708.06733)
- Too 9y agoRelated: Slight Street Sign Modifications Can Fool Machine Learning Algorithms https://news.ycombinator.com/item?id=14935120 https://news.ycombinator.com/item?id=14935120
- sumeera 9y agoМŶ ČĹÁŚŚМÁŤĔ'Ś ŚŤĔР ŚĨŚŤĔŔ МÁĶĔŚ 83 ÁŃ ĤŐÚŔ ŐŃ ŤĤĔ ĨŃŤĔŔŃĔŤ ÁŃĎ ŚĤĔ'Ś ßĔĔŃ ÚŃĔМРĹŐŶĔĎ ŦŐŔ 5 МŐŃŤĤŚ... ĹÁŚŤ МŐŃŤĤ ĤĔŔ ČĤĔČĶ ŴÁŚ 21002 ĴÚŚŤ ŴŐŔĶĨŃĞ ŐŃ ŤĤĔ ĨŃŤĔŔŃĔŤ Á ŦĔŴ ĤŐÚŔŚ РĔŔ ŴĔĔĶ.. ŚĔĔ ... https://goo.gl/89QhtU https://goo.gl/89QhtU
- fagnerbrack 9y agoThank you, an academic article without paywall
- DrNuke 9y agoFrom a workflow point of view, this is a datasets security / integrity problem, isn't it? Public and open source datasets should come with some sort of sanity check then. A best practice protocol for pre-processing private datasets / unvetted sources should also be made public and disseminated.
- m3kw9 9y agoOne way to beat back these types of attack is to have the software do a type of image filtering that averages out the adjacent pixel in a indeterminate(random) way before feeding into the NN for inference
- yodon 9y agoNot really. The single-pixel attacks are impressive because they just modify individual pixels but the concept of adversarial image generation is a broad and increasingly well studied area within neural networks. There are infinitely many ways to structure adversarial images not just single pixel defects.
- pmarreck 9y agoMaybe Marvin Minsky was right about neural nets (he’d surely be shaking his head right now, maybe dropping an “I told you so” or two...)
- NPMaxwell 9y agoThe field of Music Theory is algorithms & patterns for music. Computer music provides the opportunity for rapid iteration. If that direction were followed intensely, music theory would become a sub-topic within Psychology.
- threecoins 9y agoThis makes me feel like neural networks are nothing but glorified hash functions. Well, if you think about it, here we are just optimizing for hash collisions of similar things.
- Sharlin 9y agoThe whole point of classification algorithms, including NNs, is to map similar inputs to similar outputs, or indeed the same output, and different images to different outputs. Hash functions usually attempt to erase all information of "similarity" between inputs. However, the metric that determines what "similar" means in a NN is not necessarily what we expect it to be. But of course NNs are definitely just functions, in the strict mathematical sense. You could replace one with a large lookup table. The interesting part is the training: how to come up with the function in the first place.
- thriftwy 9y agoIt is kinda obvious that you should make multiple passes at recognition with jitter and dithering.
- deleted 9y ago[deleted]