4 ms·

This is all doable today. As an example, check out some bird photos[1] from the Visual Genome[2] project that are similar to your examples. I selected the photo

This is all doable today. As an example, check out some bird photos[1] from the Visual Genome[2] project that are similar to your examples. I selected the photos and hosted them on Imgur in hopes we don't kill Visual Genome with traffic ;) The systems to do this today are not highly efficient or without flaws but it can certainly be done. The research group I am part of, Salesforce Research (formerly MetaMind), have a model that does this "accidentally" - and there's even an example image of a bird[3]! The model is only meant to provide a caption for an image, not to segment the image into the various objects, but learns to "focus" on the bird as part of describing the image. For those particularly interested, check out the paper "Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning"[5]. Systems made specifically to segment an image into objects would obviously do far better. For an example of that, check out "CRF as RNN - Semantic Image Segmentation Live Demo"[4]. There are many more systems of this style floating about. [1]: http://imgur.com/a/1UPnn http://imgur.com/a/1UPnn [2]: http://visualgenome.org/ http://visualgenome.org/ [3]: http://imgur.com/vbX8NNZ http://imgur.com/vbX8NNZ [4]: http://www.robots.ox.ac.uk/~szheng/crfasrnndemo http://www.robots.ox.ac.uk/~szheng/crfasrnndemo [5]: https://arxiv.org/abs/1612.01887 https://arxiv.org/abs/1612.01887

TheOtherHobbes 9y ago
I think you underestimate the problem, which is not to get an output that says "Bird", but one that says "Specific breed of bird." Human experts can get enough clues from the bird shape and the context to do that in the sample photos. I doubt your captioning system can. This is a good example of a standard problem in ML - underestimating the complexity of the problem domain. You could argue that your system only needs to do the simpler task to be useful, and that's likely true. But if the goal is to approach human expert levels of classification, it needs to improve by at least a few levels. I suspect getting it there would run into some interesting performance constraints, and possibly some theoretical issues too.
- mnx 9y ago
  The problem posed in the xkcd is "Check if the photo is of a bird", not identify the bird in question. As far as identifying the bird species, that would probably be harder, because I'm guessing there's very few human experts who could reliably do that across a wide spectrum of species, and without knowing the context of the photo.
- deleted 9y ago
  [deleted]
- nl 9y ago
  No, ML is very, very good at doing breeds. See, for example https://arxiv.org/pdf/1603.06765.pdf https://arxiv.org/pdf/1603.06765.pdf which gets 88.9% accuracy on the Stanford Dogs dataset, and 84.3 on the Caltech Birds dataset. These are way better than anything a non-expert human can do. For example, it can distinguish between the Rhinoceros Auklet and the Parakeet Auklet. I'm not sure what expert performance is, but around 94% is where humans top out on most tasks. Also, the parent poster knows what they are talking about: https://www.semanticscholar.org/author/Stephen-Merity/3375440 https://www.semanticscholar.org/author/Stephen-Merity/337544...
  - chrisan 9y ago
    How far away is ML from identifying everything in a picture? For example if we could take a photo of Noah's Ark loading up every animal? Do you just loop through each NN you have on each species?
    - rjtavares 9y ago
      This seems to be what you are looking for: https://code.facebook.com/posts/561187904071636/segmenting-and-refining-images-with-sharpmask/ https://code.facebook.com/posts/561187904071636/segmenting-a...
    - nl 9y ago
      A single NN can predict more than one class of object. The ImageNet competition has 20,000 classes. There's also image segmentation as another poster has pointed to. In the case of FB face tagging, they'd have learn an embedding space for faces, and when a new image comes in they'd place it in the embedding space along with all the person's connections and find the nearest neighbors. See https://arxiv.org/abs/1503.03832 https://arxiv.org/abs/1503.03832 or the implementation https://cmusatyalab.github.io/openface/ https://cmusatyalab.github.io/openface/

TheOtherHobbes 9y ago

I think you underestimate the problem, which is not to get an output that says "Bird", but one that says "Specific breed of bird." Human experts can get enough clues from the bird shape and the context to do that in the sample photos. I doubt your captioning system can. This is a good example of a standard problem in ML - underestimating the complexity of the problem domain. You could argue that your system only needs to do the simpler task to be useful, and that's likely true. But if the goal is to approach human expert levels of classification, it needs to improve by at least a few levels. I suspect getting it there would run into some interesting performance constraints, and possibly some theoretical issues too.

mnx 9y ago
The problem posed in the xkcd is "Check if the photo is of a bird", not identify the bird in question. As far as identifying the bird species, that would probably be harder, because I'm guessing there's very few human experts who could reliably do that across a wide spectrum of species, and without knowing the context of the photo.
deleted 9y ago
[deleted]
nl 9y ago
No, ML is very, very good at doing breeds. See, for example https://arxiv.org/pdf/1603.06765.pdf https://arxiv.org/pdf/1603.06765.pdf which gets 88.9% accuracy on the Stanford Dogs dataset, and 84.3 on the Caltech Birds dataset. These are way better than anything a non-expert human can do. For example, it can distinguish between the Rhinoceros Auklet and the Parakeet Auklet. I'm not sure what expert performance is, but around 94% is where humans top out on most tasks. Also, the parent poster knows what they are talking about: https://www.semanticscholar.org/author/Stephen-Merity/3375440 https://www.semanticscholar.org/author/Stephen-Merity/337544...
- chrisan 9y ago
  How far away is ML from identifying everything in a picture? For example if we could take a photo of Noah's Ark loading up every animal? Do you just loop through each NN you have on each species?
  - rjtavares 9y ago
    This seems to be what you are looking for: https://code.facebook.com/posts/561187904071636/segmenting-and-refining-images-with-sharpmask/ https://code.facebook.com/posts/561187904071636/segmenting-a...
  - nl 9y ago
    A single NN can predict more than one class of object. The ImageNet competition has 20,000 classes. There's also image segmentation as another poster has pointed to. In the case of FB face tagging, they'd have learn an embedding space for faces, and when a new image comes in they'd place it in the embedding space along with all the person's connections and find the nearest neighbors. See https://arxiv.org/abs/1503.03832 https://arxiv.org/abs/1503.03832 or the implementation https://cmusatyalab.github.io/openface/ https://cmusatyalab.github.io/openface/