15 ms·

How Adversarial Attacks Work

300 points by lainon 9y ago

deleted 9y ago
[deleted]
trott 9y ago
Very recently, it was found that changing a single, carefully placed, pixel is enough to confuse NN classifiers. https://arxiv.org/abs/1710.08864 https://arxiv.org/abs/1710.08864
- dontreact 9y ago
  For 32 by 32 images
  - shpx 9y ago
    1/(32*32) = 0.1%
yters 9y ago
Ah, this is why AlphaGo won't release their source code or models. If all ML algorithms can be fooled so trivially, this shows the human mind is not an ML algorithm.
- j2kun 9y ago
  Take care, the original authors were being overly broad when they implied "all ML algorithms" can be fooled in this way. Not that I disagree with your conclusion. Just the hypothesis.
  - yters 9y ago
    All ML algorithms are either inconsistent or incomplete, which we know a priori. Insofar as ML tries to be complete, it will be inconsistent, as we see with adversarial examples.
- maltalex 9y ago
  > this shows the human mind is not an ML algorithm. not necessarily. Humans have a lot more knowledge about contextual information that today's ML often misses. For every "image" you classify you use a LOT of side data. That side data has also been acquired through learning. Even if your actually saw a 1,000,000$ check or a 200 km/h speed limit, you'd know it's probably nonsense. Based on your life experience so far, you know that there's no way that mom just gave you a 1,000,000$ check, or that the city decided to turn that small residential street into a racing track. An image classification ML algorithm doesn't know any of that.
  - no_flags 9y ago
    What about the Sylvester Stallone/Keanu Reeves example? It was just a zoomed in head shot. That seems like an example where humans do a lot better without help from any additional context.
- munificent 9y ago
  I think your logic is: 1. All ML algorithms can be fooled trivially. 2. The human mind cannot be fooled trivially. 3. Therefore, the human mind is not an ML algorithm. But claim number 2 is clearly wrong. Human minds are trivially fooled. Here's one: http://www.jimonlight.com/wp-content/uploads/2012/02/Parallel-Lines.gif http://www.jimonlight.com/wp-content/uploads/2012/02/Paralle... This is exactly what an optical illusion is.
  - schoen 9y ago
    Yes, when I read "Humans have no problem correctly interpreting adversarial examples, so there must be a way to do it automatically" in the article I thought "aren't optical illusions adversarial examples for human perception?" and also "maybe we just haven't found the other kinds of adversarial examples that work against human perception because we can't show humans sufficiently large numbers of examples and measure the humans' responses accurately enough". But the optical illusion example seems very strong because if you say that human perception includes abilities to classify things, there are many examples just like your link that show cases where the classification routinely goes wrong. It seems possible to me that there are as-yet undiscovered adversarial examples in human perception that we simply don't have any feasible way to search for, and may never be able to construct practical examples of. (It would probably be extremely unnerving to experience one in real life.)
    - yters 9y ago
      Optical illusions are reverse adversarial examples to those who think the human mind is a deep neural network.
  - danharaj 9y ago
    Cool illusion; I figured it out and so will most humans. I don't think the claim that these sorts of illusions are of the same nature as, say, single pixel attacks against neural networks is justified. On the other hand, it is also true that it's not clear whether such attacks on our current crop of primitive neural networks will work on more advanced autonomous systems that regulate themselves.
joe_the_user 9y ago
"Recent studies by Google Brain have shown that any machine learning classifier can be tricked to give incorrect predictions" -- That has to be an overly broad statement (edit: it might true if you say "neural nets" or something specific instead). I would assume they mean any standard deep learning system and maybe any system that is more or less "generalized regression" but that couldn't be "any machine learning system", I mean one could imagine a deterministic model that couldn't be "tricked". Plus a link would be good (it's not around the quote, I don't know if it's elsewhere in the article).
- anishathalye 9y ago
  It's probably referencing Szegedy et al. 2013 (https://arxiv.org/abs/1312.6199 https://arxiv.org/abs/1312.6199).
- zitterbewegung 9y ago
  No system is 100% secure. What they mean is you can train an adversarial input that would work with a deep learning system and this can fool other machine learning systems. Or that you can create a deep learning system that would fool another (you use a GAN to make forgeries to fool your target ).
  - j2kun 9y ago
    > No [machine learning] system is 100% secure. That is what's being claimed without proof, when they meant "existing neural network systems are known to be insecure"
    - LolWolf 9y ago
      Sure, you can't claim that every machine learning system is susceptible to this attack (here's one that isn't, a model that always returns the same class on all inputs). But (apart from obvious trivialities) I suspect this is a more general problem with interesting properties—the claim the GP is making boils down to something like: a non-trivial[0] machine learning model with large parameter space is, with vanishing probability (as the number of distinct training samples increases), robust to classes of eps-bounded attacks.[1] This seems like a provable claim that is well-defined and (in a weird, handwavy 'intuitive' kind of way) likely to be true... there are just way too many parameters and too much uncertainty in the local minima that we reach when training from such classes of examples to have 'robustness'. Perhaps I'm totally wrong, though, and someone will come up with a regularizer that prevents all of this from happening, but this seems highly not obvious to me, at first glance. --- [0] Non-trivial means that, say, it has non-vanishing curvature (Fisher information, assuming the model returns a vector of probabilities) almost everywhere. I hate to be so nitpicky, but I suspect someone would soon comment asking for definitions of all of these things and that the problem isn't 'well-defined.' [1] Given a (large enough) set of inputs, the max-norm difference between at least one input and the adversarial example must be ≤eps. In other words, if we want the model to misclassify 'turtle' with 'dog', we don't just give a dog picture---the adversarial picture must look like a turtle in a mathematical sense.
eroccatlun 9y ago
Does the adversarial attack require access to the model making the predictions? It seems like the attack relies on `doping` the input with features present in a different target, or by masking features of the existing target. Could you so specifically attack a target without knowledge of its features?
dontreact 9y ago
I don’t buy any of the attacks listed here or see how the examples being imperceptible is actually a factor. If you have the ability to modify the check why not make it actually look like it’s for a 1000000 dollars (ie even to a human). If you are going to go out and replace speed limit signs to fool self driving cars, it’s probably equally dangerous whether or not the change is obvious, because if it’s way out of bounds a car won’t do it but if it isn’t then it would also fool humans. Anyone have some more realistic attacks that are specifically possible due to the imperceptible nature of the changes in the input? Most of the harm I’ve heard in examples simply comes from the fact that if you are controlling the input to an ml system, out can get it to do whatever you want even without using any of these techniques: just actually change the class of the input.
- squeaky-clean 9y ago
  > If you have the ability to modify the check why not make it actually look like it’s for a 1000000 dollars (ie even to a human). Plausible deniability? "I don't know why it took $1 million for a $100 check, look the check clearly says $100 on it, I didn't edit it. Must be a bank glitch."
  - rtkwe 9y ago
    All the deposit systems I've used recently still require you to enter the deposit amount so if you're entering 1000000 for a check that 'clearly' says 100 you lose all plausible deniability because you're actively lying to the bank at that point.
    - throwaway613834 9y ago
      Huh, I'd always wondered why they ask you to enter the number. Thanks for pointing this out!
      - rtkwe 9y ago
        Yeah it's a combo of that and also just a check on the CV that reads the checks. Mainly it's the latter since it'll be really rare for people to maliciously enter an incorrect amount.
matt4077 9y ago
> Lately, safety concerns about AI were revolving around ethics — today we are going to talk about more pressuring[sic] and real issues. Nice how this casually demeans peoples' worries. Of course, the (demonstrated) idea that ML algorithm would pick up on, and amplify, discrimination (among other things) isn't "real" to these guys. Risk-scoring of loan applications actually happens to be one of the few uses of ML in the non-tec sector, and it's incredibly likely that some of them are already denying people's application because they happen to be named "La David" and not "Emil". But maybe the authors just don't consider "ethics" to ever be a "real" problem? But "change a single pixel and the ATM gives you $1,000,000" is, apparently, a "real" and "pressing" problem.
neptvn 9y ago
> Apart from the fact that nobody wants to risk having false positives, there’s a simple argument as old as machine learning itself: whatever a human can do, a machine can be taught to do. Humans have no problem correctly interpreting adversarial examples, so there must be a way to do it automatically. So by all means, the authors (and a vast majority of researchers) seem to be confident that ML/DL is the road to AGI, hence can "solve" human intelligence (given it is computational)? For how long are we gonna drag the adage that mimicking a human (Turing test) is equal to reaching human levels of intelligence?
- moxious 9y ago
  It is not true that whatever a human can do, a machine can be taught to do. The human must have insight into HOW they do it in order to teach it, or otherwise come up with some new algorithm. There are a large class of things humans do that they don't understand the mechanics behind, and for which there also aren't algorithms. I'm not talking empathy or philosophy. How about just folding laundry. Not just one type, not in a controlled environment, but folding any laundry anywhere.
  - Pedro9009 9y ago
    It is not true that whatever a human can do, is the correct way to look at it. The fine motor skills and insight needed for many tasks will always be beyond that of a machine. A machine also has a great deal of trouble with adapting to things popping up that it has to deal with.
  - IanCal 9y ago
    > It is not true that whatever a human can do, a machine can be taught to do. The human must have insight into HOW they do it in order to teach it, or otherwise come up with some new algorithm. Why does this mean a machine cannot be taught to do it? > There are a large class of things humans do that they don't understand the mechanics behind, Sure. > and for which there also aren't algorithms. Well, we manage to encode it in our brains.
- zardo 9y ago
lazugod 9y ago
Adversarial attacks sound useful. How can we make it illegal to prevent them?
- matt4077 9y ago
  I may not completely understand the question, but I'm pretty sure "Intentionally fooling an algorithm for nefarious purposes is illegal" is both the answer, and the status quo.
mLuby 9y ago
Am I the only one who thinks "adversarial attack" is both a redundant and unhelpful name? - Redundant: Anyone who attacks you is by definition your adversary. - Unhelpful: According to the article, "designing an input in a specific way to get the wrong result from the model is called an adversarial attack." That sounds much closer to spoofing attack ("a situation in which one person or program successfully masquerades as another by falsifying data" -Wikipedia). For example, a turtle masquerades as a gun by spoofing the machine learning system by changing irrelevant visual details.
- tlb 9y ago
  Adversarial means something specific in the ML community. https://en.wikipedia.org/wiki/Adversarial_machine_learning https://en.wikipedia.org/wiki/Adversarial_machine_learning
scrooched_moose 9y ago
Is there any reason to think this would work at all in the real world? All of these "attacks" require complete control of the image being fed to the classifier. In the ATM example you don't directly load an image of the check to the computer inside the machine. You design a check in photoshop, add the noise, print it out, feed it to the machine, which takes a picture of the check. Mobile bank apps still require you to take a picture of the check so you don't have enough control there either. Similarly in the road sign example; the lighting, angle between car and sign, dirt on the sign, etc all mean the car sees a much different image than you designed. I'd think all of these steps mean the classifier gets a dramatically different image than you intend and the attack fails. There's maybe a vanishingly small probability it works when the stars align, but that could be easily mitigated by taking multiple consecutive images and looking for an odd results.
- anishathalye 9y ago
  Goodfellow et al. demonstrate that adversarial misclassification works in the physical world: https://arxiv.org/abs/1607.02533 https://arxiv.org/abs/1607.02533 -- a printed image is consistently misclassified (demo here: https://www.youtube.com/watch?v=zQ_uMenoBCk https://www.youtube.com/watch?v=zQ_uMenoBCk) Recent work by me and some friends demonstrates that physical-world adversarial examples can actually be made quite robust, and you can synthesize 3D adversarial objects as well, and make them consistently classify as a desired target class: http://www.labsix.org/physical-objects-that-fool-neural-nets/ http://www.labsix.org/physical-objects-that-fool-neural-nets...
  - scrooched_moose 9y ago
    Thanks for the links and great work! I hadn't seen any research on it making the jump to the real world yet.
  - mikepurvis 9y ago
    I wouldn't have believed but for those videos. Fascinating stuff.
- ronald_raygun 9y ago
ChuckMcM 9y ago
This weakness is one that I think will plague self driving cars as sign recognition will be key and without some ability to insure that they cannot be dangerously fooled, it will be hard to get them certified. The canonical example is to make a no left turn sign recognize as a no right turn sign and have the car go the wrong way on a one way street. Clearly there is a marketing opportunity for t-shirts that make you recognize as other things. Who doesn't want to show up in an image search for toasters on Google Images ? :-) But my current best guess on how this issue will be addressed will be with classifier diversity an voting systems. While that just moves the problem into a harder and harder to synthesize data set (something that not only is adversarial to one classifier but gives the same answer on several), I believe it will get us to the point where we can trust the level of work to defeat them is sufficiently hard to make it a non-threat.
- projectileboy 9y ago
  One possible mitigation could be the use of geocoded roadway metadata. I rented a car in France that had this - it would let me know when I was exceeding the speed limit based on my position. Not hyper-accurate, and so probably not a replacement for all road signs, but certainly good enough for many situations.
  - smogcutter 9y ago
    Was thinking of this too, but what happens when the map and "ground truth" disagree? The machine would need to be able to identify the rare circumstances when it should trust the map over its senses. If it's capable of that, this wouldn't be a problem in the first place.
    - SomeStupidPoint 9y ago
      When driving, that probably doesn't really matter: If the map and road sign disagree, just do what everyone around you is doing (ie, go with the flow) until you recover agreement. It's not perfect, but stretches out attacks to having to compromise long stretches of road and multiple vehicle types. If the cars for the next mile somewhere can find pace, the whole group can; similarly, if any kind of car can find pace, they can all use that to set. Obviously safety checks and such, but human drivers only loosely follow signs and often get behavioral cues from other cars -- why can't automated cars do the same when they get confused?
sandGorgon 9y ago
Can such methods be used to "fingerprint" proprietary datasets by tainting them ? For example, i want to make sure that my dataset is not stolen and used by someone else (Waymo?). So I taint it using an adversarial method and create a "canary test set" that will uniquely identify if my dataset has been used in some training.
- haraldurt 9y ago
  In one sense, no. You can guarantee privacy of any given input (or any subset of k inputs) by applying transfer learning of an ensemble of models trained on subsets of the training data [0][1]. This is useful if, for instance, you train on medical data and you don't want anyone to know that "John Doe, HIV+" was part of the input. If your adversary does not take such precautions, however, then your canary should work. [0] https://arxiv.org/abs/1610.05755 https://arxiv.org/abs/1610.05755 [1] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46307.pdf https://static.googleusercontent.com/media/research.google.c...
- alexcnwy 9y ago
  I believe google maps has fake streets as a canary for their maps being scraped...
  - cscheid 9y ago
    yup, in cartography these are known as trap streets
- ovi256 9y ago
  You don't even need an adversarial attack method to fingerprint or watermark a proprietary dataset. If the dataset consists of images, you could just watermark them, or mark them using steganography. The watermark would appear as random (non-targeted) noise, and would be mostly invisible to current classifiers. Please note that the noise employed by adversarial attacks is very different, it is highly targeted.
naveen99 9y ago
Don't you need access to the classifier internals to train the adversarial network ? Nobody is going to publish the network weights for a check reading machine...
- Winterflow3r 9y ago
  No, this paper by Papernot et al shows how to do blackbox attacks without knowledge of model internals. https://arxiv.org/abs/1602.02697 https://arxiv.org/abs/1602.02697
  - naveen99 9y ago
    Thanks. But they still need to be able to use the black box a brute force number of times on shady inputs and get detailed outputs during their gradient descent. Not going to be allowed on a check reader or anything sensitive.
    - haraldurt 9y ago
      Not necessarily. Adversarial examples have been shown to, for instance, be transferable across different networks with different hyperparameters (e.g., number of layers) trained on disjoint subsets of a training set [0, section 4.2]. There are more references from the paper linked by the OP. [0] https://arxiv.org/abs/1312.6199 https://arxiv.org/abs/1312.6199
      - naveen99 9y ago
        Thanks. I wonder if adversarial training helps prevent overfitting too. Could you use adversarial training to beat alphago ?
        yorwba 9y ago
        You could not, because AlphaGo is not a classifier (so it isn't well-defined what an adversarial example is) and the input space is discrete (Go board state) and you can't do ε-small perturbations (two different states differ by at least one stone).
pdeburen 9y ago
While it's true that these attacks work well on state-of-the-art models, there are defence strategies such as including adversial examples during training. Advanced defence strategies such as https://arxiv.org/abs/1705.07204 https://arxiv.org/abs/1705.07204 are robust to a wide array of attacks and achieve very competitive error rates. I'm not saying it's not a problem but there are successful defence strategies already in place for many attacks.
- LolWolf 9y ago
  > Although our models are more vulnerable to white-box FGSM samples compared to the v3adv model, ensemble adversarial training significantly increases robustness to black-box attacks that transfer FGSM samples crafted on the holdout Inception v4. So, I just train my new adversary on the "new" model that was trained on the previous adversarial examples. And now we're back to square one. I suspect the problem of adversarial attacks is a problem of high-dimensional spaces, not of training on particular samples.
  - pdeburen 9y ago
    I'm not sure how the quote supports your argument. Adversial examples generalize well accross many different classifiers. Shallow NN's can be fooled just as well, it seems to be more of a problem of linear models in general. Apparently Geoff Hintons Capsule Networks are more robust due to being "less linear" (Ian Goodfellow mentioned this in a recent talk, don't have the references now to back it up)
    - LolWolf 9y ago
      I'm not sure it's about how shallow the network is; even logistic regression can be fooled by the same techniques (e.g. 1-layer NN). That being said, maybe it does have something to do with linearity (I suspect not) or maybe it's just generally harder to deal with nonlinear functions.
drudru11 9y ago
ATMs using neural nets - lol - nope. That is a stretch.
- genericpseudo 9y ago
  Effective counterexample: the USPS. http://www.cedar.buffalo.edu/~srihari/talks/Telcordia.pdf http://www.cedar.buffalo.edu/~srihari/talks/Telcordia.pdf
d--b 9y ago
Research is needed in that area, but surely adding some noise / blurring to sample during training and during recognition should help reduce the feasibility of the attack. Isn't that the case? Adversarial attacks seem to point to overtraining in some sense.
- shmel 9y ago
  Unfortunately, it is not that simple. There are multiple ways to generate adversarial examples and simple defenses help at most against the simplest attacks. Every now and again a new preprint "one simple trick defeating the latest adversarial defense technique" appears on arxiv. I believe they are linked to the nature of DL (and ML in general) models. We try to capture very tiny manifold of natural images in the space of all possible images. We found a technique that does it well (CNNs). But by the very definition of its training we train it to output some values in a finite number of points. In the same time CNN's output outside of those points are mainly defined by the model's smoothness. We can expect that in a neighborhood of a given image the output will be roughly the same (it allows them to generalize). Far from any point of training set the model can say anything. Adversarial examples basically hint that this smoothness work well only along natural images manifold, once we step outside it is much more chaotic. Or, equivalently, the neighborhood where the CNN gives roughly the same output is a very thin "slice" that closely follows natural images manifold. Why is that? Probably there are some non-trivial topological reasons. But it is exactly what nobody understands now.
aidenn0 9y ago
ELI5: Why are adversarial attacks not preventable by adding unpredictable noise to untrusted inputs?
- Houshalter 9y ago
  The attacks are resistant to noise, or at least can made to be so. If every single input is tweaked in exactly the right direction, noise won't undo that. Most inputs will still be pointing in the adversarial direction. The noise will move some inputs back to their original position, but others will be pushed even further into adversarial territory.
  - aidenn0 9y ago
    Thanks, that makes a lot of sense.
  - SNIPERBABY 9y ago
    Good to know, thanks!
- WalterSear 9y ago
  Because they aren't noise: they just seem like noise to humans. If you add in noise, then you have to train the network to disregard that noise. And the adversarial input will then be features that occur above this noise floor you ignoring.
xstartup 9y ago
There is a huge industry for folks who are finding ways to bypass Facebook's/Google's ad moderation. They pay you if you craft ads as per their liking and it approved.
xstartup 9y ago
Well, this is getting more attention because it's important for the "ad network moderation". If Google/FB fail at moderation using these methods, they'll have to HIRE lots of humans to do it for them, which often involves contracting it to outsources like Accenture. This will put downward pressure on their billion dollar revenue. Humans are expensive but vastly effective at ad moderation.
dontreact 9y ago
Was there a follow up to this paper? NO Need to Worry about Adversarial Examples in Object Detection in Autonomous Vehicles https://arxiv.org/abs/1707.03501 https://arxiv.org/abs/1707.03501
conorcleary 9y ago
The _only_ solution is coming up with a globally-accepted open source base set of instruction for any and all AI or AI-capable code.
samat 9y ago
Do this attacks still work if you don’t control input image bit-by-bit, but rather feed it via digital camera? While using any digits camera for input in the real world there will be some blurrines and distortion. Is it possible to construct such a noise that 1) will survive camera input 2) will be undetectable with human eyes 3) will fool a ML algorithm? I do believe you are not able to control input image bit-by-bit in many real-world scenarios.
- taejo 9y ago
  As has been posted elsewhere in this thread, you can 3D print objects that are consistently misclassified when viewed through a camera from any angle: http://www.labsix.org/physical-objects-that-fool-neural-nets/ http://www.labsix.org/physical-objects-that-fool-neural-nets...
blauditore 9y ago
It seems like all adversarial examples I've seen so far are enabled by networks overfitting to small-scale features. By overfitting I mean that they give more importance to details compared to humans, who recognize much more the image as a whole. Maybe the problem could be mitigated by penalizing "direct" responses to small-scale features while training, which is not a trivial thing to do though. One approach I could think of is training with multiple altered versions of the image, e.g. various amounts of blur, noise and mean/median filters applied. Or the other way around: To be more confident about a result, scale down the image to a fraction of it's size, run detection on that and compare results. Are such techniques in use, or being researched on? I'm not too much in the loop about those topics.
- rahimnathwani 9y ago
  "training with multiple altered versions of the image, e.g. various amounts of blur, noise and mean/median filters applied" "Are such techniques in use" Yes, it's called 'data augmentation': https://medium.com/towards-data-science/deep-learning-3-more-on-cnns-handling-overfitting-2bd5d99abe5d https://medium.com/towards-data-science/deep-learning-3-more...
breakingcups 9y ago
> How is this possible? No machine learning algorithm is perfect and they make mistakes — albeit very rarely. Right.
bartkappenburg 9y ago
I tried the example from the blog post (Stallone <-> Reeves) in Rekognition (AWS service) and it came out correct: https://imgur.com/a/C0s05 https://imgur.com/a/C0s05
amelius 9y ago
Am I the only one who finds the term "Adversarial Attack" a bit funny? See [1]. [1] https://en.wikipedia.org/wiki/Pleonasm https://en.wikipedia.org/wiki/Pleonasm
erikb 9y ago
I see an SaaS opportunity here, that takes your photos, adds noise, and only then uploads it to social networks.
barrkel 9y ago
ISTM that many of the classifiers get a lot of their signal from detecting textures, and that the adversarial noise works by superimposing a different texture on the image.
acallahan 9y ago
Obscurity seems like useful security here. IIUC it shouldn't be possible to e.g. trick self-driving cars with noisy signs, unless you have a copy of the classifier to train against. Thinking about ATMs, you could train against it as a black box, repeatedly inserting different patterns of noise? But it seems probably infeasible if you need to do a lot of iterations. It also suggests that people concerned about adversarial attacks shouldn't use off-the-shelf pretrained classifiers, where attacks can be trained offline in advance. Similar to hashing algorithms and rainbow tables, maybe a practice of "salting" an off-the-shelf classifier could be effective in dodging attacks.
- 3pt14159 9y ago
  True. Also, just having unconnected systems that use different types of features / heuristics should be enough to at least pull the car over when they wildly disagree over what to do.

300 points by lainon 9y ago

deleted 9y ago

[deleted]

Very recently, it was found that changing a single, carefully placed, pixel is enough to confuse NN classifiers. https://arxiv.org/abs/1710.08864 https://arxiv.org/abs/1710.08864

dontreact 9y ago
For 32 by 32 images
- shpx 9y ago
  1/(32*32) = 0.1%

yters 9y ago

Ah, this is why AlphaGo won't release their source code or models. If all ML algorithms can be fooled so trivially, this shows the human mind is not an ML algorithm.

j2kun 9y ago
Take care, the original authors were being overly broad when they implied "all ML algorithms" can be fooled in this way. Not that I disagree with your conclusion. Just the hypothesis.
- yters 9y ago
  All ML algorithms are either inconsistent or incomplete, which we know a priori. Insofar as ML tries to be complete, it will be inconsistent, as we see with adversarial examples.
maltalex 9y ago
> this shows the human mind is not an ML algorithm. not necessarily. Humans have a lot more knowledge about contextual information that today's ML often misses. For every "image" you classify you use a LOT of side data. That side data has also been acquired through learning. Even if your actually saw a 1,000,000$ check or a 200 km/h speed limit, you'd know it's probably nonsense. Based on your life experience so far, you know that there's no way that mom just gave you a 1,000,000$ check, or that the city decided to turn that small residential street into a racing track. An image classification ML algorithm doesn't know any of that.
- no_flags 9y ago
  What about the Sylvester Stallone/Keanu Reeves example? It was just a zoomed in head shot. That seems like an example where humans do a lot better without help from any additional context.
munificent 9y ago
I think your logic is: 1. All ML algorithms can be fooled trivially. 2. The human mind cannot be fooled trivially. 3. Therefore, the human mind is not an ML algorithm. But claim number 2 is clearly wrong. Human minds are trivially fooled. Here's one: http://www.jimonlight.com/wp-content/uploads/2012/02/Parallel-Lines.gif http://www.jimonlight.com/wp-content/uploads/2012/02/Paralle... This is exactly what an optical illusion is.
- schoen 9y ago
  Yes, when I read "Humans have no problem correctly interpreting adversarial examples, so there must be a way to do it automatically" in the article I thought "aren't optical illusions adversarial examples for human perception?" and also "maybe we just haven't found the other kinds of adversarial examples that work against human perception because we can't show humans sufficiently large numbers of examples and measure the humans' responses accurately enough". But the optical illusion example seems very strong because if you say that human perception includes abilities to classify things, there are many examples just like your link that show cases where the classification routinely goes wrong. It seems possible to me that there are as-yet undiscovered adversarial examples in human perception that we simply don't have any feasible way to search for, and may never be able to construct practical examples of. (It would probably be extremely unnerving to experience one in real life.)
  - yters 9y ago
    Optical illusions are reverse adversarial examples to those who think the human mind is a deep neural network.
- danharaj 9y ago
  Cool illusion; I figured it out and so will most humans. I don't think the claim that these sorts of illusions are of the same nature as, say, single pixel attacks against neural networks is justified. On the other hand, it is also true that it's not clear whether such attacks on our current crop of primitive neural networks will work on more advanced autonomous systems that regulate themselves.

joe_the_user 9y ago

"Recent studies by Google Brain have shown that any machine learning classifier can be tricked to give incorrect predictions" -- That has to be an overly broad statement (edit: it might true if you say "neural nets" or something specific instead). I would assume they mean any standard deep learning system and maybe any system that is more or less "generalized regression" but that couldn't be "any machine learning system", I mean one could imagine a deterministic model that couldn't be "tricked". Plus a link would be good (it's not around the quote, I don't know if it's elsewhere in the article).

anishathalye 9y ago
It's probably referencing Szegedy et al. 2013 (https://arxiv.org/abs/1312.6199 https://arxiv.org/abs/1312.6199).
zitterbewegung 9y ago
No system is 100% secure. What they mean is you can train an adversarial input that would work with a deep learning system and this can fool other machine learning systems. Or that you can create a deep learning system that would fool another (you use a GAN to make forgeries to fool your target ).
- j2kun 9y ago
  > No [machine learning] system is 100% secure. That is what's being claimed without proof, when they meant "existing neural network systems are known to be insecure"
  - LolWolf 9y ago
    Sure, you can't claim that every machine learning system is susceptible to this attack (here's one that isn't, a model that always returns the same class on all inputs). But (apart from obvious trivialities) I suspect this is a more general problem with interesting properties—the claim the GP is making boils down to something like: a non-trivial[0] machine learning model with large parameter space is, with vanishing probability (as the number of distinct training samples increases), robust to classes of eps-bounded attacks.[1] This seems like a provable claim that is well-defined and (in a weird, handwavy 'intuitive' kind of way) likely to be true... there are just way too many parameters and too much uncertainty in the local minima that we reach when training from such classes of examples to have 'robustness'. Perhaps I'm totally wrong, though, and someone will come up with a regularizer that prevents all of this from happening, but this seems highly not obvious to me, at first glance. --- [0] Non-trivial means that, say, it has non-vanishing curvature (Fisher information, assuming the model returns a vector of probabilities) almost everywhere. I hate to be so nitpicky, but I suspect someone would soon comment asking for definitions of all of these things and that the problem isn't 'well-defined.' [1] Given a (large enough) set of inputs, the max-norm difference between at least one input and the adversarial example must be ≤eps. In other words, if we want the model to misclassify 'turtle' with 'dog', we don't just give a dog picture---the adversarial picture must look like a turtle in a mathematical sense.

eroccatlun 9y ago

Does the adversarial attack require access to the model making the predictions? It seems like the attack relies on `doping` the input with features present in a different target, or by masking features of the existing target. Could you so specifically attack a target without knowledge of its features?

dontreact 9y ago

I don’t buy any of the attacks listed here or see how the examples being imperceptible is actually a factor. If you have the ability to modify the check why not make it actually look like it’s for a 1000000 dollars (ie even to a human). If you are going to go out and replace speed limit signs to fool self driving cars, it’s probably equally dangerous whether or not the change is obvious, because if it’s way out of bounds a car won’t do it but if it isn’t then it would also fool humans. Anyone have some more realistic attacks that are specifically possible due to the imperceptible nature of the changes in the input? Most of the harm I’ve heard in examples simply comes from the fact that if you are controlling the input to an ml system, out can get it to do whatever you want even without using any of these techniques: just actually change the class of the input.

squeaky-clean 9y ago
> If you have the ability to modify the check why not make it actually look like it’s for a 1000000 dollars (ie even to a human). Plausible deniability? "I don't know why it took $1 million for a $100 check, look the check clearly says $100 on it, I didn't edit it. Must be a bank glitch."
- rtkwe 9y ago
  All the deposit systems I've used recently still require you to enter the deposit amount so if you're entering 1000000 for a check that 'clearly' says 100 you lose all plausible deniability because you're actively lying to the bank at that point.
  - throwaway613834 9y ago
    Huh, I'd always wondered why they ask you to enter the number. Thanks for pointing this out!
    - rtkwe 9y ago
      Yeah it's a combo of that and also just a check on the CV that reads the checks. Mainly it's the latter since it'll be really rare for people to maliciously enter an incorrect amount.

matt4077 9y ago

> Lately, safety concerns about AI were revolving around ethics — today we are going to talk about more pressuring[sic] and real issues. Nice how this casually demeans peoples' worries. Of course, the (demonstrated) idea that ML algorithm would pick up on, and amplify, discrimination (among other things) isn't "real" to these guys. Risk-scoring of loan applications actually happens to be one of the few uses of ML in the non-tec sector, and it's incredibly likely that some of them are already denying people's application because they happen to be named "La David" and not "Emil". But maybe the authors just don't consider "ethics" to ever be a "real" problem? But "change a single pixel and the ATM gives you $1,000,000" is, apparently, a "real" and "pressing" problem.

neptvn 9y ago

> Apart from the fact that nobody wants to risk having false positives, there’s a simple argument as old as machine learning itself: whatever a human can do, a machine can be taught to do. Humans have no problem correctly interpreting adversarial examples, so there must be a way to do it automatically. So by all means, the authors (and a vast majority of researchers) seem to be confident that ML/DL is the road to AGI, hence can "solve" human intelligence (given it is computational)? For how long are we gonna drag the adage that mimicking a human (Turing test) is equal to reaching human levels of intelligence?

moxious 9y ago
It is not true that whatever a human can do, a machine can be taught to do. The human must have insight into HOW they do it in order to teach it, or otherwise come up with some new algorithm. There are a large class of things humans do that they don't understand the mechanics behind, and for which there also aren't algorithms. I'm not talking empathy or philosophy. How about just folding laundry. Not just one type, not in a controlled environment, but folding any laundry anywhere.
- Pedro9009 9y ago
  It is not true that whatever a human can do, is the correct way to look at it. The fine motor skills and insight needed for many tasks will always be beyond that of a machine. A machine also has a great deal of trouble with adapting to things popping up that it has to deal with.
- IanCal 9y ago
  > It is not true that whatever a human can do, a machine can be taught to do. The human must have insight into HOW they do it in order to teach it, or otherwise come up with some new algorithm. Why does this mean a machine cannot be taught to do it? > There are a large class of things humans do that they don't understand the mechanics behind, Sure. > and for which there also aren't algorithms. Well, we manage to encode it in our brains.
zardo 9y ago

lazugod 9y ago

Adversarial attacks sound useful. How can we make it illegal to prevent them?

matt4077 9y ago
I may not completely understand the question, but I'm pretty sure "Intentionally fooling an algorithm for nefarious purposes is illegal" is both the answer, and the status quo.

mLuby 9y ago

Am I the only one who thinks "adversarial attack" is both a redundant and unhelpful name? - Redundant: Anyone who attacks you is by definition your adversary. - Unhelpful: According to the article, "designing an input in a specific way to get the wrong result from the model is called an adversarial attack." That sounds much closer to spoofing attack ("a situation in which one person or program successfully masquerades as another by falsifying data" -Wikipedia). For example, a turtle masquerades as a gun by spoofing the machine learning system by changing irrelevant visual details.

tlb 9y ago
Adversarial means something specific in the ML community. https://en.wikipedia.org/wiki/Adversarial_machine_learning https://en.wikipedia.org/wiki/Adversarial_machine_learning

scrooched_moose 9y ago

Is there any reason to think this would work at all in the real world? All of these "attacks" require complete control of the image being fed to the classifier. In the ATM example you don't directly load an image of the check to the computer inside the machine. You design a check in photoshop, add the noise, print it out, feed it to the machine, which takes a picture of the check. Mobile bank apps still require you to take a picture of the check so you don't have enough control there either. Similarly in the road sign example; the lighting, angle between car and sign, dirt on the sign, etc all mean the car sees a much different image than you designed. I'd think all of these steps mean the classifier gets a dramatically different image than you intend and the attack fails. There's maybe a vanishingly small probability it works when the stars align, but that could be easily mitigated by taking multiple consecutive images and looking for an odd results.

anishathalye 9y ago
Goodfellow et al. demonstrate that adversarial misclassification works in the physical world: https://arxiv.org/abs/1607.02533 https://arxiv.org/abs/1607.02533 -- a printed image is consistently misclassified (demo here: https://www.youtube.com/watch?v=zQ_uMenoBCk https://www.youtube.com/watch?v=zQ_uMenoBCk) Recent work by me and some friends demonstrates that physical-world adversarial examples can actually be made quite robust, and you can synthesize 3D adversarial objects as well, and make them consistently classify as a desired target class: http://www.labsix.org/physical-objects-that-fool-neural-nets/ http://www.labsix.org/physical-objects-that-fool-neural-nets...
- scrooched_moose 9y ago
  Thanks for the links and great work! I hadn't seen any research on it making the jump to the real world yet.
- mikepurvis 9y ago
  I wouldn't have believed but for those videos. Fascinating stuff.
ronald_raygun 9y ago

ChuckMcM 9y ago

This weakness is one that I think will plague self driving cars as sign recognition will be key and without some ability to insure that they cannot be dangerously fooled, it will be hard to get them certified. The canonical example is to make a no left turn sign recognize as a no right turn sign and have the car go the wrong way on a one way street. Clearly there is a marketing opportunity for t-shirts that make you recognize as other things. Who doesn't want to show up in an image search for toasters on Google Images ? :-) But my current best guess on how this issue will be addressed will be with classifier diversity an voting systems. While that just moves the problem into a harder and harder to synthesize data set (something that not only is adversarial to one classifier but gives the same answer on several), I believe it will get us to the point where we can trust the level of work to defeat them is sufficiently hard to make it a non-threat.

projectileboy 9y ago
One possible mitigation could be the use of geocoded roadway metadata. I rented a car in France that had this - it would let me know when I was exceeding the speed limit based on my position. Not hyper-accurate, and so probably not a replacement for all road signs, but certainly good enough for many situations.
- smogcutter 9y ago
  Was thinking of this too, but what happens when the map and "ground truth" disagree? The machine would need to be able to identify the rare circumstances when it should trust the map over its senses. If it's capable of that, this wouldn't be a problem in the first place.
  - SomeStupidPoint 9y ago
    When driving, that probably doesn't really matter: If the map and road sign disagree, just do what everyone around you is doing (ie, go with the flow) until you recover agreement. It's not perfect, but stretches out attacks to having to compromise long stretches of road and multiple vehicle types. If the cars for the next mile somewhere can find pace, the whole group can; similarly, if any kind of car can find pace, they can all use that to set. Obviously safety checks and such, but human drivers only loosely follow signs and often get behavioral cues from other cars -- why can't automated cars do the same when they get confused?

sandGorgon 9y ago

Can such methods be used to "fingerprint" proprietary datasets by tainting them ? For example, i want to make sure that my dataset is not stolen and used by someone else (Waymo?). So I taint it using an adversarial method and create a "canary test set" that will uniquely identify if my dataset has been used in some training.

haraldurt 9y ago
In one sense, no. You can guarantee privacy of any given input (or any subset of k inputs) by applying transfer learning of an ensemble of models trained on subsets of the training data [0][1]. This is useful if, for instance, you train on medical data and you don't want anyone to know that "John Doe, HIV+" was part of the input. If your adversary does not take such precautions, however, then your canary should work. [0] https://arxiv.org/abs/1610.05755 https://arxiv.org/abs/1610.05755 [1] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46307.pdf https://static.googleusercontent.com/media/research.google.c...
alexcnwy 9y ago
I believe google maps has fake streets as a canary for their maps being scraped...
- cscheid 9y ago
  yup, in cartography these are known as trap streets
ovi256 9y ago
You don't even need an adversarial attack method to fingerprint or watermark a proprietary dataset. If the dataset consists of images, you could just watermark them, or mark them using steganography. The watermark would appear as random (non-targeted) noise, and would be mostly invisible to current classifiers. Please note that the noise employed by adversarial attacks is very different, it is highly targeted.

naveen99 9y ago

Don't you need access to the classifier internals to train the adversarial network ? Nobody is going to publish the network weights for a check reading machine...

Winterflow3r 9y ago
No, this paper by Papernot et al shows how to do blackbox attacks without knowledge of model internals. https://arxiv.org/abs/1602.02697 https://arxiv.org/abs/1602.02697
- naveen99 9y ago
  Thanks. But they still need to be able to use the black box a brute force number of times on shady inputs and get detailed outputs during their gradient descent. Not going to be allowed on a check reader or anything sensitive.
  - haraldurt 9y ago
    Not necessarily. Adversarial examples have been shown to, for instance, be transferable across different networks with different hyperparameters (e.g., number of layers) trained on disjoint subsets of a training set [0, section 4.2]. There are more references from the paper linked by the OP. [0] https://arxiv.org/abs/1312.6199 https://arxiv.org/abs/1312.6199
    - naveen99 9y ago
      Thanks. I wonder if adversarial training helps prevent overfitting too. Could you use adversarial training to beat alphago ?
      - yorwba 9y ago
        You could not, because AlphaGo is not a classifier (so it isn't well-defined what an adversarial example is) and the input space is discrete (Go board state) and you can't do ε-small perturbations (two different states differ by at least one stone).

pdeburen 9y ago

While it's true that these attacks work well on state-of-the-art models, there are defence strategies such as including adversial examples during training. Advanced defence strategies such as https://arxiv.org/abs/1705.07204 https://arxiv.org/abs/1705.07204 are robust to a wide array of attacks and achieve very competitive error rates. I'm not saying it's not a problem but there are successful defence strategies already in place for many attacks.

LolWolf 9y ago
> Although our models are more vulnerable to white-box FGSM samples compared to the v3adv model, ensemble adversarial training significantly increases robustness to black-box attacks that transfer FGSM samples crafted on the holdout Inception v4. So, I just train my new adversary on the "new" model that was trained on the previous adversarial examples. And now we're back to square one. I suspect the problem of adversarial attacks is a problem of high-dimensional spaces, not of training on particular samples.
- pdeburen 9y ago
  I'm not sure how the quote supports your argument. Adversial examples generalize well accross many different classifiers. Shallow NN's can be fooled just as well, it seems to be more of a problem of linear models in general. Apparently Geoff Hintons Capsule Networks are more robust due to being "less linear" (Ian Goodfellow mentioned this in a recent talk, don't have the references now to back it up)
  - LolWolf 9y ago
    I'm not sure it's about how shallow the network is; even logistic regression can be fooled by the same techniques (e.g. 1-layer NN). That being said, maybe it does have something to do with linearity (I suspect not) or maybe it's just generally harder to deal with nonlinear functions.

drudru11 9y ago

ATMs using neural nets - lol - nope. That is a stretch.

genericpseudo 9y ago
Effective counterexample: the USPS. http://www.cedar.buffalo.edu/~srihari/talks/Telcordia.pdf http://www.cedar.buffalo.edu/~srihari/talks/Telcordia.pdf

d--b 9y ago

Research is needed in that area, but surely adding some noise / blurring to sample during training and during recognition should help reduce the feasibility of the attack. Isn't that the case? Adversarial attacks seem to point to overtraining in some sense.

shmel 9y ago
Unfortunately, it is not that simple. There are multiple ways to generate adversarial examples and simple defenses help at most against the simplest attacks. Every now and again a new preprint "one simple trick defeating the latest adversarial defense technique" appears on arxiv. I believe they are linked to the nature of DL (and ML in general) models. We try to capture very tiny manifold of natural images in the space of all possible images. We found a technique that does it well (CNNs). But by the very definition of its training we train it to output some values in a finite number of points. In the same time CNN's output outside of those points are mainly defined by the model's smoothness. We can expect that in a neighborhood of a given image the output will be roughly the same (it allows them to generalize). Far from any point of training set the model can say anything. Adversarial examples basically hint that this smoothness work well only along natural images manifold, once we step outside it is much more chaotic. Or, equivalently, the neighborhood where the CNN gives roughly the same output is a very thin "slice" that closely follows natural images manifold. Why is that? Probably there are some non-trivial topological reasons. But it is exactly what nobody understands now.

aidenn0 9y ago

ELI5: Why are adversarial attacks not preventable by adding unpredictable noise to untrusted inputs?

Houshalter 9y ago
The attacks are resistant to noise, or at least can made to be so. If every single input is tweaked in exactly the right direction, noise won't undo that. Most inputs will still be pointing in the adversarial direction. The noise will move some inputs back to their original position, but others will be pushed even further into adversarial territory.
- aidenn0 9y ago
  Thanks, that makes a lot of sense.
- SNIPERBABY 9y ago
  Good to know, thanks!
WalterSear 9y ago
Because they aren't noise: they just seem like noise to humans. If you add in noise, then you have to train the network to disregard that noise. And the adversarial input will then be features that occur above this noise floor you ignoring.

xstartup 9y ago

There is a huge industry for folks who are finding ways to bypass Facebook's/Google's ad moderation. They pay you if you craft ads as per their liking and it approved.

xstartup 9y ago

Well, this is getting more attention because it's important for the "ad network moderation". If Google/FB fail at moderation using these methods, they'll have to HIRE lots of humans to do it for them, which often involves contracting it to outsources like Accenture. This will put downward pressure on their billion dollar revenue. Humans are expensive but vastly effective at ad moderation.

dontreact 9y ago

Was there a follow up to this paper? NO Need to Worry about Adversarial Examples in Object Detection in Autonomous Vehicles https://arxiv.org/abs/1707.03501 https://arxiv.org/abs/1707.03501

conorcleary 9y ago

The _only_ solution is coming up with a globally-accepted open source base set of instruction for any and all AI or AI-capable code.

samat 9y ago

Do this attacks still work if you don’t control input image bit-by-bit, but rather feed it via digital camera? While using any digits camera for input in the real world there will be some blurrines and distortion. Is it possible to construct such a noise that 1) will survive camera input 2) will be undetectable with human eyes 3) will fool a ML algorithm? I do believe you are not able to control input image bit-by-bit in many real-world scenarios.

taejo 9y ago
As has been posted elsewhere in this thread, you can 3D print objects that are consistently misclassified when viewed through a camera from any angle: http://www.labsix.org/physical-objects-that-fool-neural-nets/ http://www.labsix.org/physical-objects-that-fool-neural-nets...

blauditore 9y ago

It seems like all adversarial examples I've seen so far are enabled by networks overfitting to small-scale features. By overfitting I mean that they give more importance to details compared to humans, who recognize much more the image as a whole. Maybe the problem could be mitigated by penalizing "direct" responses to small-scale features while training, which is not a trivial thing to do though. One approach I could think of is training with multiple altered versions of the image, e.g. various amounts of blur, noise and mean/median filters applied. Or the other way around: To be more confident about a result, scale down the image to a fraction of it's size, run detection on that and compare results. Are such techniques in use, or being researched on? I'm not too much in the loop about those topics.

rahimnathwani 9y ago
"training with multiple altered versions of the image, e.g. various amounts of blur, noise and mean/median filters applied" "Are such techniques in use" Yes, it's called 'data augmentation': https://medium.com/towards-data-science/deep-learning-3-more-on-cnns-handling-overfitting-2bd5d99abe5d https://medium.com/towards-data-science/deep-learning-3-more...

breakingcups 9y ago

> How is this possible? No machine learning algorithm is perfect and they make mistakes — albeit very rarely. Right.

bartkappenburg 9y ago

I tried the example from the blog post (Stallone <-> Reeves) in Rekognition (AWS service) and it came out correct: https://imgur.com/a/C0s05 https://imgur.com/a/C0s05

amelius 9y ago

Am I the only one who finds the term "Adversarial Attack" a bit funny? See [1]. [1] https://en.wikipedia.org/wiki/Pleonasm https://en.wikipedia.org/wiki/Pleonasm

erikb 9y ago

I see an SaaS opportunity here, that takes your photos, adds noise, and only then uploads it to social networks.

barrkel 9y ago

ISTM that many of the classifiers get a lot of their signal from detecting textures, and that the adversarial noise works by superimposing a different texture on the image.

acallahan 9y ago

Obscurity seems like useful security here. IIUC it shouldn't be possible to e.g. trick self-driving cars with noisy signs, unless you have a copy of the classifier to train against. Thinking about ATMs, you could train against it as a black box, repeatedly inserting different patterns of noise? But it seems probably infeasible if you need to do a lot of iterations. It also suggests that people concerned about adversarial attacks shouldn't use off-the-shelf pretrained classifiers, where attacks can be trained offline in advance. Similar to hashing algorithms and rainbow tables, maybe a practice of "salting" an off-the-shelf classifier could be effective in dodging attacks.

3pt14159 9y ago
True. Also, just having unconnected systems that use different types of features / heuristics should be enough to at least pull the car over when they wildly disagree over what to do.