14 ms·
How to trick a neural network into thinking a panda is a vulture
- schiffern 11y agoI wonder if simulated saccading would address this problem? Rank the image by entropy, and re-run the neural network centered on those points with a radially-increasing gaussian blur applied to the subimage (approximating the sensitivity of the human visual field).
- extrapolate 11y agoGreat read! Time to troll some of my machine learning co-workers. "Can you classify this image of a panda? Oh, curious, it thinks it is a vulture..."
- thomasahle 11y agoI don't think it'll nessesarily work, if you take an adversial example for one network and plugs it into another.
- jvns 11y agosurprisingly, if you read the paper, the adversarial examples actually do generalize to some extent if you try them on another network!
- Houshalter 11y agoHere is the table from the original paper. Note this is for MNIST, not more complicated datasets like Imagenet. http://i.imgur.com/fJ35PTc.png http://i.imgur.com/fJ35PTc.png It's kind of confusing, but table 2 shows what percent of these adversarial images trained on one networked worked on another. It varies quite a bit, and many networks aren't similar enough to each other for it to work reliably. But there is definitely some degree of generalization.
- rrmm 11y agoThere are tons of anecdotes i remember from AI classes about neural nets learning something other than what you expected them to learn from your training set. For example, one story involved training a classifier to recognize an overhead image with tanks vs without. It turns out it ended up learning which days were sunny and which were overcast. This sort of thing happens when training people from examples too: From the mundane cases in school, to the AA587 crash in Queens NYC.
- cbr 11y ago> ended up learning which days were sunny and which were overcast. The earliest source I can find for this is https://neil.fraser.name/writing/tank/ https://neil.fraser.name/writing/tank/ but it says it "might be apocryphal".
- cbr 11y agoActually, there's also the 1992 paper "What Artificial Experts Can and Cannot Do" [1] which gives this anecdote and calls it a "legend". [1] http://www.jefftk.com/dreyfus92.pdf http://www.jefftk.com/dreyfus92.pdf
- rrmm 11y agoNice sleuthing! I don't even recall where I first heard this story myself. The paper was an interesting read and still largely applicable even now in this 3rd or 4th coming of NN's.
- cbr 11y agoExpanded this into a post: http://www.jefftk.com/p/detecting-tanks http://www.jefftk.com/p/detecting-tanks
- guard-of-terra 11y agoNeural net can see some subtle pattern invisible to human eye, decide it is very representative, and make a guess based on that. Pre-processing seems to be an answer, normalize the image, blank out areas obviously uninteresting to human eye, add dithering.
- argonaut 11y agoDefine "uninteresting." Also, it kind of defeats the purpose of neural networks to do substantial feature engineering like that.
- RodericDay 11y agoI'm not in computer science or anything but... is there like a big ML vs. ML-skeptic divide in the field? Where could I read about it?
- guard-of-terra 11y agoSomething human can't see is uninteresting. Fractional values of pixel color is one thing. Our human eyes have a lot of filters (hardware and software-based) before recognition takes place.
- sp332 11y agoThe neural network should capture all the behavior of the relevant filters as well.
- guard-of-terra 11y agoI'm not sure. Human brain uses dedicated facilities for this. You won't win much by making every neural network learn stuff from scratch that can be done once, good.
- orting 11y agoThere is a lot of stuff the human eye cant see that is very interesting. One of the challenges in medical imaging is getting accurate labelling of images. For many labellings we see large inter- and intra-observer variability. We have both the problem that humans see something that is not interesting and miss something that is interesting. I currently work on estimating emphysema extent in CT lung scans. Emphysema can be very diffuse and it is not possible to label individual pixels, so instead we try to learn the local emphysema pattern from a global label. Neural networks are interesting for this problem because the learn the features, but it is also a "problem" because the features might not make physically sense, which could make it hard to transfer the model and convince clinicians that they should use it.
- pmalynin 11y agohttps://news.ycombinator.com/item?id=10742390 https://news.ycombinator.com/item?id=10742390
- natch 11y agoCool! So now I have this Python notebook running (I guess) on docker, and it tells me: [I 08:37:21.591 NotebookApp] Writing notebook server cookie secret to /.local/share/jupyter/runtime/notebook_cookie_secret [I 08:37:21.757 NotebookApp] Serving notebooks from local directory: /neural-nets [I 08:37:21.758 NotebookApp] 0 active kernels [I 08:37:21.759 NotebookApp] The IPython Notebook is running at: http://0.0.0.0:8888/ [I 08:37:21.759 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). What next? Should I be looking for a python notebook tutorial, or a docker tutorial, or both? The author lists some commands to run, but it isn't clear where to type those commands. All I see is a terminal with the above output.
- jre 11y agoOpen a browser and navigate to your docker container's :8888 port. That would be localhost:9990 if you used the docker commandline given in the repo.
- zenlikethat 11y agoOr :9990 on the Docker Machine VM's IP address if you're using Machine.
- avian 11y agoI don't know anything about Docker, but to use iPython Notebook, you need to open it in a browser. The command line starts a local web server. It says it is running at "http://0.0.0.0:8888/" http://0.0.0.0:8888/" in your case (which sounds invalid). Usually, Notebook opens the browser for you automatically when it starts. I guess Docker interferes with that.
- antsar 11y ago0.0.0.0 is valid. It means the server is listening on all available addresses (as opposed to, say, only 127.0.0.1)
- blazespin 11y ago
- dnautics 11y agothis really suggests that while neural networks are good, they aren't doing what we are doing when we classify images.
- shoo 11y agoi have a vague suspicion, without evidence, that many of these systems are trained to overfit their training sets horrendously. i won't claim that all networks overfit the data, but i suspect the methods tend to be prone to overfitting more than other methods that have far less parameters to optimise. this doesn't really matter in order to generate a bunch of fun pictures and visualisations.
- blazespin 11y agoMaybe, it's hard to say. I remember when my kids were really young they used to see really random things in clouds and sand or whatever - just like a neural net would.
- kuschku 11y agoThere’s even a name (and a subreddit) for the phenomenon: https://www.reddit.com/r/pareidolia https://www.reddit.com/r/pareidolia It shows that a human brain is just a neural network with decades of training.
- dnautics 11y agoconvnets are based off of the structure in the visual cortex, so there are likely some low-level similarities. It's the high-level processing that I doubt.
- Houshalter 11y agoWell we don't know that for sure. No one has been able to run backpropagation through the human brain, and find the exact right set of inputs that would fool us.
- deleted 11y ago[deleted]
- blazespin 11y agoFor some reason this reminds me of the man who mistook his wife for a hat. https://en.wikipedia.org/wiki/The_Man_Who_Mistook_His_Wife_for_a_Hat https://en.wikipedia.org/wiki/The_Man_Who_Mistook_His_Wife_f...
- spikels 11y agoMake me think of this and optical illusions. http://www.ritsumei.ac.jp/~akitaoka/index-e.html http://www.ritsumei.ac.jp/~akitaoka/index-e.html
- IshKebab 11y agoI like the visualisations on this page: http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ (Especially "The Easy Way Out" one. It suggests that because neural networks cannot properly separate the classes, the often mush them together in the input-space so the "panda" region of image-space has loads of other classes speckled though it. If you purposely find those speckles you can make it think that a panda is a vulture. That blog mentions adding smoothness constraints, so that you can't suddenly go from panda to vulture with a tiny change in an image. But I wonder if an easier solution (ok, hack) is just to add different noise to the image 10 times or so, run it through the network and then combine the results. Essentially you'd be doing Monte Carlo sampling of the image space around the input image. Or kind of blurring the image space. Just an idea anyway. I don't really know what I'm talking about.
- Houshalter 11y agoFrom the papers, training on images with random noise in them didn't help. But intentionally creating these adversarial images and training on them did help. And even improved the performance of the net in general.
- IshKebab 11y agoI don't mean training on images with random noise - what about evaluating the net on images with random noise?
- fundamental 11y agoMy understanding from reading a few 'adversarial example' papers is that it's also possible to talk about the classifier as having a poor margin between classes. An acceptable output classifier for the convolutional net would be to place the decision boundary arbitrarily close to the known examples, but this doesn't make much sense given how large the semantic differences actually are. There are multiple options to try to enhance the distance to the decision boundary (such as the adversarial examples referenced in the post), but I think the recent work from microsoft (I think it's published Dec 2015) on replacing pooling layers in the convolutional neural net with something analogous random forests might be the best option so far. More or less each decision tree will end up pushing values to 0/1 in the non-linear region, which mitigates some of the concern about overly linear systems and it places the decision boundary at a somewhat arbitrary location between classes. In aggregate when these locations are combined the resulting classifier has a larger margin without explicitly needing adversaries. So instead of sampling the image space, you're effectively sampling the classifier space. If you want, I can dig up the citation, but searching for deep neural nets and random forests should get you to the paper all the same.
- awqrre 11y agoCould processing the image before and after adding a small blur to the image detect this kind of thing?
- sp332 11y agoYou would lose information from the texture of the image that way. It's already downscaled to 224 pixels square, so you'd really be restricting the amount of data available to the function.
- awqrre 11y agoof course you would loose some information but it is probably not enough to mis-identify most objects..
- gcb0 11y agoThis is a great example of why computer scientists get a bad rep. zero understanding of something so simple, laid out as black magic. And a lot more time spent fiddling with the thing as a toddler playing with a toy than it would be required to read two or three articles that explained it correctly.
- jvns 11y agowhich articles would you suggest?
- lordnacho 11y agoQuite interesting. Sounds like he's subtly modifying the images at the points where the various classes are sensitive. I don't think any humans are going to be fooled by this procedure, so what algorithm do people use to classify? There must be other algos out there that are more plausible as explanations for what humans (and animals) use?
- deleted 11y ago[deleted]
- argonaut 11y agoThis implies human eyes are extremely robust. But the truth is our eyes can be tricked very easily too. For what it's worth, of all the intelligent things the human brain performs, the visual system is the system that is closest to what neural nets are doing.
- wyager 11y agoThere are a number of obvious tricks that humans can use and simple NN classifiers can't. For example, humans have access to facts, which allow for better use of context information. E.g. I know that pandas live in trees, and that the queen wears a crown. It does appear that humans do better with limited training data. I have probably seen fewer pandas than the NN in the article, but I can identify them more reliably. It probably comes down to humans' superior ability to generalize training examples.
- eli_gottlieb 11y ago>I don't think any humans are going to be fooled by this procedure, so what algorithm do people use to classify? There must be other algos out there that are more plausible as explanations for what humans (and animals) use? Personally, it seems very plausible to me that humans and animals use generative modeling (which was behind the "human-level concept learning" paper published this month) rather than discriminative (like typical neural networks).
- tinco 11y agoThey way I understand is that humans simply use more than just a simple single neural network. Humans do more than just compare arrays of pixels to other arrays of pixels we've seen before. We have neurons that distinguish shapes, we have neurons that distinguish faces, we have neurons that distinguish textures, etc. So for a human the question is more like: Does it have a shape like a Panda, does it have a face like a panda, does it have fur like a panda, and does it have coloring similar to a panda? Then it probably is a panda. I also don't believe all of these systems are trained neural networks, but rather some of them are evolved instincts, hardwired and (near) unchangeable during a lifetime. Perfected over millions of years.
- ohblahitsme 11y agoWow this post is fantastic! Great read Julia!
- sundarurfriend 11y agoI should probably begin reading author names when starting to read articles - knowing this was written by Julia Evans would have clued me in on how fun a read this was going to be, and motivated me to cut down on the open-tab-procrastination hours! > So now we’ve seen the network do a correct thing, and we’ve seen it make an adorable mistake by accident (the queen is wearing a shower cap ). This is such a Julia sentence, including the suddenly appropriate emoji, it should probably have clued me in. Edit: HN is breaking my heart with its anti-emoji stance, so (re)read the article to see the quote in its full glory.
- chestervonwinch 11y agoI'm not sure if this provides any insight, but I think it's interesting to visualize this type of thing for the much simpler case of a two-class problem in two dimensions: http://i.imgur.com/NdzdH5j.png http://i.imgur.com/NdzdH5j.png On the left are the training vectors, color-coded by label; the background color-codes the probability output by the learned network at each point in the plane. On the right is the gradient field of the network's output corresponding to the blue class. The gradient field shows, at any point, the local direction of greatest increase toward the blue class.
- eli_gottlieb 11y agoInteresting. So most of the training points for Blue were in the "bulge" near where they were surrounded by the Red class, but at almost every point, the network believes that moving further "deeper" into the Blue "zone" increases the "blueness".
- deleted 11y ago[deleted]
- deleted 11y ago[deleted]
- Qantourisc 11y agoWhat happens if you retrain the network with these bugs ? (As in tell the vulture is not a panda ?)
- jvns 11y agothe network gets smarter! (you can read the paper to find out more =D)