6 ms·

Adversarial Examples That Fool Both Computer Vision and Time-Limited Humans

• dontreact 8y ago
  Does this mean we can stop seeing adversarial examples as a deep fundamental flaw of deep neural networks? Seems like human system experiences them as well!
  • carlmr 8y ago
    We know human systems experience them as well. But we can design roads and other things so that they don't trick us too much. We have very little intuitive design understanding of what can trick robots. And in a lot of cases this is also something how you can hack an AI system if you understand what it misperceives.
    • jon_richards 8y ago
      As xkcd points out, you don't have to worry too much about people trying to trick self-driving systems with visual hacks. https://xkcd.com/1958/ https://xkcd.com/1958/
      • carlmr 8y ago
        Ok, fair point about the hacking, but unintentionally misleading things are also hard to estimate.
      • menudo 8y ago
        The XKCD comic strip makes for a useful and shallow zinger, amid casual banter, but the real world actually faces two or three new layers of complexity, which actually reassert the problem it attempts to dispose of. 1. The global reach of networked telecommunications permits a small quantity of sociopathic murderers to operate from beyond jurisdictions that can reach them, and also assists in destroying evidence of their interference. 2. Computational power enables force multiplication, such that even just one sociopathic murderer could exploit software flaws across millions of vehicles, simultaneously. 3. Some software exploits will work against self driving cars, which could never work against an ordinary person, and of course, vice versa, but not so much via remote control at a distance, when people are the operators, while we still lack electronic interfaces to our central nervous system.
        • jon_richards 8y ago
          >visual hacks We were discussing "tricking" the computer vision component of a self-driving car, not getting software access. That's still a concern, but it's an entirely different set of security requirements that we already face in planes and existing cars.
          • menudo 8y ago
            Except XKCD wasn't.
            • jon_richards 8y ago
              To me, >painting fake lines on the road, or dropping a cutout of a pedestrian onto a highway sounds like >"tricking" the computer vision component of a self-driving car, not getting software access.
  • saagarjha 8y ago
    Aren't optical illusions just adversarial examples for people?
    • aetherson 8y ago
      No, and I wish people would stop that meme. Humans can obviously be tricked, in a variety of ways. But adversarial images take advantage of the fact that image-recognizing neural networks do not fit their image recognition into a full fledged understanding of the world like we do. So a few pixels here and there can make a truck look like a panda and the algorithm never says, "But wait, pandas are mostly black and white and this is mostly yellow," or, "But I don't see legs anywhere, or ears." Optical illusions mostly don't cause high level image misclassifications. To the extent that they are anything similar, they're the reverse: using our general world understanding to cause glitches in our information processing, such as cases where you think something is darker or lighter than it is, or bigger or smaller, or bent or straight. Those are your mind applying rules that are based on "how the world usually appears at a high level" to an image where those rules do not in fact apply.
      • saagarjha 8y ago
        I’m no machine learning expert, but it seems to me that neural networks just don’t really work like people do, as much as people would like to claim that they’ve created something that works like the human brain.
        • aetherson 8y ago
          Sure. So, then, why did you posit a direct parallel between optical illusions and adversarial images?
          • saagarjha 8y ago
            Because that’s another thing I can now mention when people tell me neural networks are like the human brain, since this is an example they often like to bring up.
  • mannykannot 8y ago
    This does not take the issue off the table, as far as practical applications of neural-network object recognition and scene analysis are concerned. Firstly, it leaves open the issue of an additional category of images for which neural networks alone very confidently misidentify things. Secondly, the time limitation, while being a perfectly valid aspect of the experiment, means that parity has not yet been achieved in these cases, either.
  • tmalsburg2 8y ago
    In the experiment, humans showed a minor drop in accuracy (<10%) after seeing the stimulus for 63ms. The presentation was so short that humans made a considerable amount of mistakes even with the non-adversarial examples. In contrast, neural networks get adversarial stimuli consistently wrong even though they are allowed to fully process them. The results also do not prove that the underlying mechanisms are the same in humans and neural networks.
  • andrewflnr 8y ago
    It doesn't decrease the actual set of attacks you can run against a DNN, so no. If anything it increases the scope of the security vulnerability.
  • andbberger 8y ago
    No
• tomcam 8y ago
  Aren’t we all time-limited humans?
  • aneeqdhk 8y ago
    On a long enough timeline the survival rate for everyone drops to zero.
• zkms 8y ago
  Speaking of similarity between CV and human visual systems; it was kinda striking to me just how psychedelic/"trippy" the "universal adversarial perturbations" (see page 5 in https://arxiv.org/pdf/1610.08401.pdf https://arxiv.org/pdf/1610.08401.pdf) look. Indeed, the look of visual hallucination "form constants" is intimately related to the machinery in the visual cortex that detects edges/contours/surfaces: https://www.math.uh.edu/~dynamics/reprints/papers/nc.pdf https://www.math.uh.edu/~dynamics/reprints/papers/nc.pdf and such visual hallucinations can be elicited without any drugs or physiological interventions/defects -- just via diffuse flickering light: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3182860/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3182860/
  • shoo 8y ago
    There's a video demo of the first paper here: https://www.youtube.com/watch?v=jhOu5yhe0rc https://www.youtube.com/watch?v=jhOu5yhe0rc i was trying to build a bit of intuition about this paper by considering a trivial case where there are two classes, and the classifier is linear, in very low dimension. consider the following trivial example: https://en.wikipedia.org/wiki/Linear_classifier#/media/File:Svm_separating_hyperplanes.png https://en.wikipedia.org/wiki/Linear_classifier#/media/File:... in this example, the decision boundary from different classifiers is shown as H_1, H_2, H_3. The "universal" perturbation for each of the three classifiers would be a small vector normal to the each classifier's decision boundary. This paper defines "universal" perturbation with respect to the choice of input from the population of inputs, but each "universal" perturbation is optimised specifically to target a single model (aka classifier). both H_1 and H_2 do a reasonable job of separating the two classes, but the H_1 decision boundary with the smaller margin is more vulnerable to misclassification if inputs are perturbed by a small vector normal to its decision boundary. You can imagine translating the input space a bit to the right -- this would result in H_1 misclassifying say 2 out of the 18 data points shown, whereas H_2 (the SVM generated decision boundary with maximal margin) classifier would not experience any errors.
• rahimnathwani 8y ago
  The interesting data are in "Table Supp. 1" at the bottom of page 14. I opened the paper hoping to see some examples of images that look to me like one thing on first glance, and something else on closer inspection. The best image is the one with the spider on a blurred-snake background, and that's not going to trick anyone who looks at it for more than a second. The humans were shown each image for either 63ms or 71ms. That's 1-2 frames of a movie. So whilst the result is important, it's not as surprising as you might expect.
  • gameswithgo 8y ago
    It reminds me of academic papers on graphics, where there is often a 50x50 pixel black and white image that looks like it was faxed back and forth 3 times. heh
  • gxigzigxigxi 8y ago
    To be fair to the computers, that’s about the speed we train image classifiers to run at, if not faster. It seems likely that if we were willing to tolerate a second or two of latency we might be able to devise architectures that are able to see through some of the images that confuse current architectures.
    • Aic1kuir 8y ago
      It's not just the time. Computers don't have higher reasoning that can cross-check the results with expectations from a more complex world model. So by just flashing images I guess you only activate some earlier parts of the visual processing machinery. So that's quite fair to the computers since their NNs only do primitive pattern checking without reasoning too.
    • lisper 8y ago
      The problem is not that we're not willing to tolerate latency. The problem is that the model of how neural networks are trained is completely different from how humans learn to see. When a neural net is trained, it is shown a static image, weights are tweaked until the output is correct, and then it is shown a completely different static image and the process is repeated. Neural net learning is iterative, discrete, and supervised. Human visual learning, by contrast, is continuous and largely unsupervised. We don't see snapshots, we see continuously varying images. Furthermore, we actively interact with the world by manipulating objects and shifting our gaze, and that information is also incorporated into our visual learning. Finally, humans have very advanced feature detectors built in to our brains by evolution. We don't learn to see cats, we have cat-detectors built in to our brains by our DNA, which learned to detect cats because that was a useful skill in our ancestral environment, when cats were a lot bigger and could eat us. We do learn that the thing that our cat-detectors detect is called a "cat", but we don't "learn" what a cat (or a human) looks like. That's built in to our brain wiring. (There are some things that we do learn what they look like, like cars, which obviously didn't exist in the ancestral environment. That's why all humans can tell the difference between a cat and a dog, but not everyone can identify whether something is a Honda or a Toyota.) The point is: the process that humans go through when they learn is completely different than the process that contemporary neural nets go through. No one has yet come up with a theory that combines all of the features of human learning into an implementable algorithm. It will surely happen eventually, but there are at least a few more conceptual breakthroughs that will need to happen. Minor tweaks to back-propagation won't do it.
• dalbasal 8y ago
  Interesting. Anyone in the space have some meta/context to share? I may have this wrong but it seems the authors are very interested in a particular subset of classification errors common to machines and humans. What is that subsets of "transferable" mistakes are trying to find? What does the existence of a particular subset of "adversarial example" (sneakily doctored images) tell us either about human or machine brains?
  • mannykannot 8y ago
    This is what the authors say: "A rigorous investigation of the above question creates an opportunity both for machine learning to gain knowledge from neuroscience, and for neuroscience to gain knowledge from machine learning."
• ardy42 8y ago
  I'm looking forward to when these are integrated into Recaptcha...