5 ms·

What I learned from competing against a ConvNet on ImageNet

32 points by Houshalter 12y ago

dave_sullivan 12y ago
Based on the sample of images I worked on, the GoogLeNet classification error turned out to be 6.8% (the error on the full test set of 100,000 images is 6.7%). My own error in the end turned out to be 5.1%, approximately 1.7% better. To put this in perspective, the task is basically "given a picture, identify the correct class out of 1000 categories" (it gets into specific breeds of dog, for instance). Humans turn out to miss some of these, and neural networks are now nearing human level performance. It's fun to note that about 4 years ago I performed a similar (but much quicker and less detailed) human classification accuracy analysis on CIFAR-10. This was back when the state of the art was at 77% by Adam Coates, and my own accuracy turned out to be 94%. I think the best ConvNets now get about 92%. Things are moving quickly in this field. (My personal) ILSVRC 2014 TLDR: 50% more teams. 50% improved classification and detection. ConvNet ensembles all over the place. Google team wins.
bainsfather 12y ago
To be clear - this is a human viewing the images, and only narrowly beating the neural net. Furthermore, the human spent some time 'training' himself on the images, plus he used a neural net to reduce the possible answer space - he didn't come to it cold. "Then I organized a labeling party of intense labeling effort only among the (expert labelers) in our lab. Then I developed a modified interface that used GoogLeNet predictions to prune the number of categories from 1000 to only about 100. It was still too hard - people kept missing categories and getting up to ranges of 13-15% error rates. In the end I realized that to get anywhere competitively close to GoogLeNet, it was most efficient if I sat down and went through the painfully long training process and the subsequent careful annotation process myself."
mdda 12y ago
There are some good TLDRs here. But the actual article includes a link to a UI with the actual ImageNet decision task created by the author. You can compete against the winning ConvNet too..
brotchie 12y ago
Go to the demonstration labeling interface here: http://cs.stanford.edu/people/karpathy/ilsvrc/ http://cs.stanford.edu/people/karpathy/ilsvrc/ Hit the "Use hard course". On the left hit "Show google prediction" consider the answers for a few seconds, then hit "Show answer". My reaction to the Google predictions is "yeah, that's reasonable". Often my reaction to the actual true label is "wtf, that's not obvious at all." I get the feeling the Google image net produces better tags than the validation set labels :)
ap22213 12y ago
This is very impressive. I hadn't realized that image classification was so good. I'm not sure I understand the results though. On the demonstration site, there's a group of images that apparently failed to be classified. But, when I tried it this morning, google's prediction shows a correct classification, for some of them. Anyway, where is this service? It would be extremely valuable to developers, even in it's current state.

32 points by Houshalter 12y ago

Based on the sample of images I worked on, the GoogLeNet classification error turned out to be 6.8% (the error on the full test set of 100,000 images is 6.7%). My own error in the end turned out to be 5.1%, approximately 1.7% better. To put this in perspective, the task is basically "given a picture, identify the correct class out of 1000 categories" (it gets into specific breeds of dog, for instance). Humans turn out to miss some of these, and neural networks are now nearing human level performance. It's fun to note that about 4 years ago I performed a similar (but much quicker and less detailed) human classification accuracy analysis on CIFAR-10. This was back when the state of the art was at 77% by Adam Coates, and my own accuracy turned out to be 94%. I think the best ConvNets now get about 92%. Things are moving quickly in this field. (My personal) ILSVRC 2014 TLDR: 50% more teams. 50% improved classification and detection. ConvNet ensembles all over the place. Google team wins.

bainsfather 12y ago

To be clear - this is a human viewing the images, and only narrowly beating the neural net. Furthermore, the human spent some time 'training' himself on the images, plus he used a neural net to reduce the possible answer space - he didn't come to it cold. "Then I organized a labeling party of intense labeling effort only among the (expert labelers) in our lab. Then I developed a modified interface that used GoogLeNet predictions to prune the number of categories from 1000 to only about 100. It was still too hard - people kept missing categories and getting up to ranges of 13-15% error rates. In the end I realized that to get anywhere competitively close to GoogLeNet, it was most efficient if I sat down and went through the painfully long training process and the subsequent careful annotation process myself."

mdda 12y ago

There are some good TLDRs here. But the actual article includes a link to a UI with the actual ImageNet decision task created by the author. You can compete against the winning ConvNet too..

brotchie 12y ago

Go to the demonstration labeling interface here: http://cs.stanford.edu/people/karpathy/ilsvrc/ http://cs.stanford.edu/people/karpathy/ilsvrc/ Hit the "Use hard course". On the left hit "Show google prediction" consider the answers for a few seconds, then hit "Show answer". My reaction to the Google predictions is "yeah, that's reasonable". Often my reaction to the actual true label is "wtf, that's not obvious at all." I get the feeling the Google image net produces better tags than the validation set labels :)

ap22213 12y ago

This is very impressive. I hadn't realized that image classification was so good. I'm not sure I understand the results though. On the demonstration site, there's a group of images that apparently failed to be classified. But, when I tried it this morning, google's prediction shows a correct classification, for some of them. Anyway, where is this service? It would be extremely valuable to developers, even in it's current state.