4 ms·

Surpassing Human-Level Performance on ImageNet Classification [pdf]

53 points by fchollet 12y ago

svantana 12y ago
Interesting, I've long wondered why parametric nonlinearities are not used more often. It adds very little to the overall parameter count (the number of units is most often dwarfed by the number of connections), but it should increase expressibility a lot (e.g. adaptively using soft or hard nonlinearities). Taken to its extreme, I've been toying with the idea of combining DNNs and Genetic Programming - using a large number of arbitrary symbolic expressions with high connectivity and many layers.
- fchollet 12y ago
  Genetic programming is an inefficient search method, and will require many evaluations of the cost function to optimize anything. In the case of DCNNs, evaluating an architecture can keep a modern GPU busy for days, so genetic algorithms are pretty much out of the question. I think an easy way to improve our models in the short term is to make more of the parameters we use be learnable: the parameters of the non-linearity are a good place to start with, and another would be the parameters of the data augmentation transformations. One could consider that learned data augmentation schemes implement a form of guided visual attention.
dwiel 12y ago
It seems that this might mean that ImageNet is becoming less useful a benchmark dataset. Some of the images have labels which I would have never guessed myself, and not because I don't know the difference between two types of stingray, but because I would have never said that the topic of the image was a seatbelt. It would be interesting to know how many errors are actually due to outputting a label which does exist in the image, but isn't in the labeled truth data and how many are plainly not in the image.
- fchollet 12y ago
  The accuracy reported is top-5 accuracy, meaning that a model is considered correct on a test image if it includes the expected label in its top 5 predicted labels. This does mitigate the multi-object issue quite a bit.
  - dwiel 12y ago
    Most of the examples cited in the paper that their algorithm got wrong, I would also get 'wrong.' I don't think I would guess restaurant for any of the images with that label, I might have gotten the middle spotlight and I might have gotten the first letter opener right, but I'm not sure. How do you explain that their performance is better than human? Is it in the obscure examples?
    - greeneggs 12y ago
      On the other hand, some of their correct images have questionable captions as well. For example, they labeled the geyser picture correctly, but their top labels also included "sandbar", "breakwater" and "leatherback turtle". A better scoring function, perhaps including hierarchies to account for the very vague "restaurant" photos and very specific dog breed photos, might be helpful. Otherwise, it seems like we might be overfitting to the peculiarities of this dataset.
- AndrewKemendo 12y ago
  Interesting. I typically am annoyed when people move the goalposts like this on AI but in my research it seems like the ImageNet library is annotating recursively through the various automation sets and not actually founded on human classification. Is that accurate?
  - dwiel 12y ago
    That is a good question. I went and read the about page at imagenet [1], but it wasn't completely obvious what the source is. One thing to note though is that the context as described there is "1000 images to illustrate each word." This doesn't exactly match with: "words which a person would most likely label an image." Check out the page on team sport for example [2]. Would you ever label one of those team sport? Or what about this one labeled 'fresh bean' [3]. It is not hard to find hard examples here [4]. [1]: http://www.image-net.org/about-overview [2]: http://www.image-net.org/synset?wnid=n00887544 [3]: http://www.image-net.org/nodes/8/07727578/5b/5bcde1a26fd46c485ca7df589e57ba2ded9cb789.thumb [4]: http://www.image-net.org/synset
    - AndrewKemendo 12y ago
      I asked the question because it seems like their library has too many images for manual input and I never found a good answer on what the seed set was on the site. I don't think the question should be "would you label these..." but moreso "is description 'X' an accurate representation of the image." In the latter case I say yes to most of the examples I find. The former question severely limits the set of possibilities to whomever is the one classifying it. I know there has been a ton of research done on this stuff but to me the context threshold for image classification seems difficult to bound because humans might be looking for something very specific in a photo, to the point of impossibly difficult to annotate. For example take this photo: http://farm1.static.flickr.com/222/497812075_d412946eef.jpg http://farm1.static.flickr.com/222/497812075_d412946eef.jpg I can think of a million annotations for it: - 9 men - Island - Trees near water - Rocks in trees - Backwards hat - Blue water - Rowing - Oar splashing - Nice day - Warm weather - White boat - Saddleback mountain etc... So how would you rank those in terms of what the picture is really showing? The only thing I can think is of popularity, which makes the most sense to me in terms of how we natively classify - but it doesn't cover boundary cases or super narrow specificity.
Animats 12y ago
That's an impressive result, considering how simple the algorithm really is. (The learning algorithm isn't obvious, but it's not a lot of code.) Can this algorithm be run in reverse, to generate an image from the network? That's been done with one of the other deep neural net classifiers. "School bus" came out as a yellow blob with horizontal black lines. An interesting question is whether there's a bias in the data set because humans composed the pictures, and humans like to take pictures of certain things. (Cats are probably over-represented.) Images taken by humans tend to have a primary subject, and that subject is usually roughly centered in the images. It might be useful to test against a data set taken from Google StreetView images, which lack such a composition bias.
- dEnigma 12y ago
  Can this algorithm be run in reverse, to generate an image from the network? That's been done with one of the other deep neural net classifiers. "School bus" came out as a yellow blob with horizontal black lines. Do you have a link for that, sounds interesting (nothing turned up in a quick google search) edit: I found something similar to what you were talking about, is this[1] what you meant? [1]http://www.evolvingai.org/fooling http://www.evolvingai.org/fooling
  - Houshalter 12y ago
    That's the paper he's referring to. They used another NN to generate images, and selected the ones that the first NN predicted to be school buses the most.

53 points by fchollet 12y ago

Interesting, I've long wondered why parametric nonlinearities are not used more often. It adds very little to the overall parameter count (the number of units is most often dwarfed by the number of connections), but it should increase expressibility a lot (e.g. adaptively using soft or hard nonlinearities). Taken to its extreme, I've been toying with the idea of combining DNNs and Genetic Programming - using a large number of arbitrary symbolic expressions with high connectivity and many layers.

fchollet 12y ago
Genetic programming is an inefficient search method, and will require many evaluations of the cost function to optimize anything. In the case of DCNNs, evaluating an architecture can keep a modern GPU busy for days, so genetic algorithms are pretty much out of the question. I think an easy way to improve our models in the short term is to make more of the parameters we use be learnable: the parameters of the non-linearity are a good place to start with, and another would be the parameters of the data augmentation transformations. One could consider that learned data augmentation schemes implement a form of guided visual attention.

dwiel 12y ago

It seems that this might mean that ImageNet is becoming less useful a benchmark dataset. Some of the images have labels which I would have never guessed myself, and not because I don't know the difference between two types of stingray, but because I would have never said that the topic of the image was a seatbelt. It would be interesting to know how many errors are actually due to outputting a label which does exist in the image, but isn't in the labeled truth data and how many are plainly not in the image.

fchollet 12y ago
The accuracy reported is top-5 accuracy, meaning that a model is considered correct on a test image if it includes the expected label in its top 5 predicted labels. This does mitigate the multi-object issue quite a bit.
- dwiel 12y ago
  Most of the examples cited in the paper that their algorithm got wrong, I would also get 'wrong.' I don't think I would guess restaurant for any of the images with that label, I might have gotten the middle spotlight and I might have gotten the first letter opener right, but I'm not sure. How do you explain that their performance is better than human? Is it in the obscure examples?
  - greeneggs 12y ago
    On the other hand, some of their correct images have questionable captions as well. For example, they labeled the geyser picture correctly, but their top labels also included "sandbar", "breakwater" and "leatherback turtle". A better scoring function, perhaps including hierarchies to account for the very vague "restaurant" photos and very specific dog breed photos, might be helpful. Otherwise, it seems like we might be overfitting to the peculiarities of this dataset.
AndrewKemendo 12y ago
Interesting. I typically am annoyed when people move the goalposts like this on AI but in my research it seems like the ImageNet library is annotating recursively through the various automation sets and not actually founded on human classification. Is that accurate?
- dwiel 12y ago
  That is a good question. I went and read the about page at imagenet [1], but it wasn't completely obvious what the source is. One thing to note though is that the context as described there is "1000 images to illustrate each word." This doesn't exactly match with: "words which a person would most likely label an image." Check out the page on team sport for example [2]. Would you ever label one of those team sport? Or what about this one labeled 'fresh bean' [3]. It is not hard to find hard examples here [4]. [1]: http://www.image-net.org/about-overview [2]: http://www.image-net.org/synset?wnid=n00887544 [3]: http://www.image-net.org/nodes/8/07727578/5b/5bcde1a26fd46c485ca7df589e57ba2ded9cb789.thumb [4]: http://www.image-net.org/synset
  - AndrewKemendo 12y ago
    I asked the question because it seems like their library has too many images for manual input and I never found a good answer on what the seed set was on the site. I don't think the question should be "would you label these..." but moreso "is description 'X' an accurate representation of the image." In the latter case I say yes to most of the examples I find. The former question severely limits the set of possibilities to whomever is the one classifying it. I know there has been a ton of research done on this stuff but to me the context threshold for image classification seems difficult to bound because humans might be looking for something very specific in a photo, to the point of impossibly difficult to annotate. For example take this photo: http://farm1.static.flickr.com/222/497812075_d412946eef.jpg http://farm1.static.flickr.com/222/497812075_d412946eef.jpg I can think of a million annotations for it: - 9 men - Island - Trees near water - Rocks in trees - Backwards hat - Blue water - Rowing - Oar splashing - Nice day - Warm weather - White boat - Saddleback mountain etc... So how would you rank those in terms of what the picture is really showing? The only thing I can think is of popularity, which makes the most sense to me in terms of how we natively classify - but it doesn't cover boundary cases or super narrow specificity.

Animats 12y ago

That's an impressive result, considering how simple the algorithm really is. (The learning algorithm isn't obvious, but it's not a lot of code.) Can this algorithm be run in reverse, to generate an image from the network? That's been done with one of the other deep neural net classifiers. "School bus" came out as a yellow blob with horizontal black lines. An interesting question is whether there's a bias in the data set because humans composed the pictures, and humans like to take pictures of certain things. (Cats are probably over-represented.) Images taken by humans tend to have a primary subject, and that subject is usually roughly centered in the images. It might be useful to test against a data set taken from Google StreetView images, which lack such a composition bias.

dEnigma 12y ago
Can this algorithm be run in reverse, to generate an image from the network? That's been done with one of the other deep neural net classifiers. "School bus" came out as a yellow blob with horizontal black lines. Do you have a link for that, sounds interesting (nothing turned up in a quick google search) edit: I found something similar to what you were talking about, is this[1] what you meant? [1]http://www.evolvingai.org/fooling http://www.evolvingai.org/fooling
- Houshalter 12y ago
  That's the paper he's referring to. They used another NN to generate images, and selected the ones that the first NN predicted to be school buses the most.