The progress that has been made in the past 5 years has been amazing. No one would have predicted superhuman performance on Imagenet in 2016. Hell no one predic

The progress that has been made in the past 5 years has been amazing. No one would have predicted superhuman performance on Imagenet in 2016. Hell no one predicted it would be even close for much simpler datasets like CIFAR-10, which is just low res images of 10 types of objects. This is amazing progress, don't let the AI Effect ruin it (https://en.wikipedia.org/wiki/AI_effect https://en.wikipedia.org/wiki/AI_effect). Second you can't measure progress without a quantitative benchmark. Feeding a net random images from a different dataset, and then just noticing it makes some mistakes, is not scientific. Sure, I agree, Imagenet has been beaten and we need something better to compare with humans. We need bigger and harder datasets. We need more interesting tasks than classification. We need to work more on video than static images. And researchers are working on this. It's not going to happen overnight, but if the current rate of progress continues, it won't be that long. Also I question if this focus on machine vision is actually that productive. Originally the developments in vision, were generalizing to many other domains. But now they are increasingly focused on little tricks and optimizations that only apply on that specific task. I don't think it's contributing towards general AI any more. The human brain has evolved to do vision well. It probably uses a huge number of tricks and optimizations to do as well as it does. NNs may eventually get that good, but it's interesting they can do so well without being so highly task specialized. This makes them very general and applicable to many other kinds of problems. Lastly, half the problem is just computing resources. The biggest nets are still roughly comparable to insect brains (more synapses, but fewer neurons.) It's really amazing that we can get such good results with such underpowered computers. Much better machine vision might be possible if we had more computing power to train with. Training on big datasets, high resolution images, and especially video, can be really expensive. >My point here is different: notice that the mistakes that those models make are completely ridiculous to humans. They are not off by some minor degree, they are just totally off. I wonder if the algorithms think the mistakes humans make are equally ridiculous? Hinton once found some crazy errors NNs made, and then pointed out that the image actually does kind of look like that thing, if you squint. >In my next post in a few days I will go deeper into the problems of deep nets and analyse the so called adversarial examples. These special stimuli reveal a lot about how convolutional nets work and what their limitations are. This is a super overblown issue. It's been shown that every machine learning algorithm is vulnerable to adversarial examples. Especially linear models, NNs are actually more resistant to it. We don't know that humans aren't vulnerable to them - no one's ever opened up a human brain and backpropagated to the inputs. Adversarial examples are astronomically unlikely to occur by chance