7 ms·

Where we see shapes, AI sees textures

98 points by gdubs 7y ago

personjerry 7y ago
This shouldn't be news. Most computer vision NNs are CNNs where the convolutions are basically translating the image into small "textures". "AI sees textures" yeah because that's how we present data to it!
- hammock 7y ago
  I wonder what would happen if we fed it SVGs instead of JPGs. (Shape files)
  - tastroder 7y ago
    At that point it's less a task in the context of image processing and more one of graph learning / language processing, depending on how you want to formulate it.
    - wongarsu 7y ago
      Human vision certainly contains a component where we do 3d reconstruction and adjust our perception based on that. Many optical illusions depend on this step. If we want more human like machine vision then having passes in image processing that deal with more abstract data sounds like a great idea
  - mirekrusin 7y ago
    I don't think those networks can work with variable inputs, they use fixed size images, svg describes paths that highly vary.
    - abrichr 7y ago
      That's true for convolutional networks. However, recurrent architectures aren't similarly limited, e.g SketchRNN: https://github.com/tensorflow/magenta/blob/master/magenta/models/sketch_rnn/README.md https://github.com/tensorflow/magenta/blob/master/magenta/mo...
CharlesW 7y ago
I'm a little confused by the assertion of "artificial intelligence's preference for texture over shape". The article makes it sound like an intrinsic issue with AI, when to my understanding it's just an outcome of popular image classification algorithms. Couldn't one create classifiers that depend more heavily on object shapes?
- swagasaurus-rex 7y ago
  I'd guess animals have a lot of compute dedicated to finding out the shape based off of various visual cues: Lighting changes indicating edges Gradients indicating a smooth curve Texture maps that discern between matte, smooth, translucent, often in depth enough to determine the texture material itself. Reflectivity Changes in reflectivity when reorienting oneself or the object Binocular Vision Parallax effect between frames A large database matching objects to memories and experiences Many other techniques for tracking and seeing movement. In the above, Parallax and Binocular vision, and consulting one's memory are the only ways to determine the sizes of the objects being looked at. It's why those optical illusions tiny/big rooms work so easily; neither we nor CV algorithms would be able to discern the sizes of objects without prior memory of such objects or by walking around it.
the_seraphim 7y ago
My question is how do we then deal with the situation where a group of cats is making up the shape of an elephant? a good AI will eventually need to be able to identify this subtle difference, it is an elephant, made of cats, not a cat and not an elephant.
udayrddy 7y ago
So, a person with a gray texture of elephant skin shirt is classified as elephant ?!
- liuliu 7y ago
  Yes, maybe. But it is at least unclear whether is inherently CNN or not (many people would vote for no). ImageNet training data is heavily texture dependent, therefore, the trained model leaning towards texture information more. Some recent papers did style transfer to randomize textures in order to train the model to focus more on the shape, the reported results seem good.
  - w_t_payne 7y ago
    Exactly. I would also add (although without strong evidence to support my assertion) that it seems to be more a function of the training data than of the network architecture.
- Sharlin 7y ago
  One of the most archetypal failure modes of current NN-based classifiers is, for example, happily reporting seeing a spotted feline given a picture of a couch with a leopard pattern.
  - pmontra 7y ago
    Well, if I'm given the task to enumerate what I see in that image maybe I would also add the feline label. But I would add the picture_of attribute, if that's a thing.
    - Sharlin 7y ago
      I don’t think today’s classifiers are anywhere sophisticated enough to have a concept of something vs. a picture of something, a type of use–mention distinction, or ceci n’est un pipe if you will. The point was that people think they trained their classifier to recognize leopards, but in actuality it just learned to recognize the leopard coat pattern.
ryu2k2 7y ago
Just a temporary restriction until we start training AI in robots by interaction with the real world.
w_t_payne 7y ago
Depends on the training data. These networks are correlation-finding machines, and will simply latch onto the simplest correlation possible that produces the correct results. Only by explicitly controlling the correlations in the training data can you force the network to focus on the attributes that you wish. This is why domain-randomization (such as the approach that NVIDIA is taking) helps support generalization -- it's a very direct and efficient form of regularization, removing correlations that do not generalize, forcing the network to work harder to find correlations that do generalize. This also has the added benefit of being something that we can tie back to requirements and physical properties, an important ingredient in making these systems understood and safe.
jcranendonk 7y ago
My take on this: if we want vision ML to succeed at recognition in the same way as humans, perhaps we need to pre-process and present visual information in the same way as the human vision system? As far as I'm aware, we get a lot of info from our eyes about lines and orientation that assists in recognizing shapes. I'm not well-informed about the current state of visual recognition DL, perhaps someone who is can tell us more about whether that approach makes sense.
- tveita 7y ago
  When you train a deep convolutional neural network, the first couple of layers appear to take on this role, detecting simple features like edges and textures, which the higher layers build upon to see more complex objects. For example https://www.researchgate.net/figure/Visualization-of-example-features-of-eight-layers-of-a-deep-convolutional-neural_fig2_279068412 https://www.researchgate.net/figure/Visualization-of-example..., where you can see (somewhat, if you zoom in) that layer 1 neurons are interested in very simple features, like strong horizontal edges, or particular gradients.
quico 7y ago
Yann LeCun (Chief AI @ FB) puts this article in perspective: https://www.facebook.com/yann.lecun/posts/10156068737942143 https://www.facebook.com/yann.lecun/posts/10156068737942143 TLDR: The use of texture is inherent to the ImageNet dataset and not to deep learning / ConvNet. Training on less-textured versions of ImageNet drives the ConvNet to focus more on shape.
- hooloovoo_zoo 7y ago
  Sure, but just because you can coerce an algorithm into doing something different does not mean the algorithm itself does not have fundamental tendencies.
- p0nce 7y ago
  Yes but what the NN exchange between layers is fundamentally textures, not shape descriptions right ?
modeless 7y ago
There is a misunderstanding here. It's not the algorithm that emphasizes textures over shapes, it's the dataset. Texture is a more informative feature than shape in the datasets used. "AI" can detect shapes just fine; simply use a different dataset and you'll get a different result with the same algorithm. I mean, just look at the classic MNIST: it's all shape, no texture, and neural nets work great.
- Majromax 7y ago
  > Texture is a more informative feature than shape in the datasets used. But the insight is that although both features are informative, classical training approaches result in networks that over-fit on texture information and ignore shape information. Table 2 of the discussed paper finds that joint training with "Stylized ImageNet" (IN images after a style transfer) and classic IN followed by a fine-tuning pass on IN results in improved classification accuracy over IN training alone.
  - skgoa 7y ago
    Which proves exactly what the top level comment claimed: the distribution of examples in the dataset was incorrect for the learning task.
    - yarg 7y ago
      Not necessarily. Textures are high frequency low abstraction data forms, shapes are low frequency high abstraction data forms. The former are easier to learn for multiple reasons: The nature of higher frequency data means that you have more samples. (There are far more furry textures in a picture of a cat than there are toes, paws, legs etc.) Low abstraction data forms are also learnt in earlier layers of the network - more abstract forms are resolved through the understanding of how less abstract forms group together, making the low abstraction forms a prerequisite to more in depth learning.
- anbop 7y ago
  One thing that’s interesting about textures is that humans have an incredible sensitivity to it by touch. You can detect the difference between a flat surface and a surface with angstrom-magnitude features.
stcredzero 7y ago
I can't wait for the Hollywood version of this idea, à la the Infrared viewer from Predator or the machine's eye view from Terminator.
jpalomaki 7y ago
Babies use their hands to feel shapes. Maybe brain works hard to make the connection between the inputs provided through touch and vision. Understanding shapes/outlines is useful for manipulating objects and avoiding obstacles. Maybe the object recognition develops a bit later and builds on top of how brains already understand the world.
- scotty79 7y ago
  I think babies also have pretty crappy sharpness of vision. Maybe that's on purpose to help with initial learning how to see...
  - mkl 7y ago
    It seems they reach healthy adult sharpness by 6 months [1]. By the time they are sitting and starting to move they are certainly extremely good at finding tiny bits of fluff etc. to put in their mouths! [1] https://en.wikipedia.org/wiki/Infant_visual_development https://en.wikipedia.org/wiki/Infant_visual_development
Havoc 7y ago
I thought that's why edge detection preprocessing etc is in use? Doesn't quite seem like a break through insight to me?
jonplackett 7y ago
Does this have anything to do with AIs always learning from 2D photos when we learn from binocular vision so every image contains a lot more ‘shape’ information.
- esotericn 7y ago
  It seems like that's fairly important to me. Consider that even if you're monocular, you're still able to determine parallax by moving your head around to determine depth.
- anewhnaccount2 7y ago
  Not only this, but we when we see things even when we are still our eyes/head move a small amount. Shape information is preserved in these multiple perspectives relatively more than texture information.

98 points by gdubs 7y ago

This shouldn't be news. Most computer vision NNs are CNNs where the convolutions are basically translating the image into small "textures". "AI sees textures" yeah because that's how we present data to it!

hammock 7y ago
I wonder what would happen if we fed it SVGs instead of JPGs. (Shape files)
- tastroder 7y ago
  At that point it's less a task in the context of image processing and more one of graph learning / language processing, depending on how you want to formulate it.
  - wongarsu 7y ago
    Human vision certainly contains a component where we do 3d reconstruction and adjust our perception based on that. Many optical illusions depend on this step. If we want more human like machine vision then having passes in image processing that deal with more abstract data sounds like a great idea
- mirekrusin 7y ago
  I don't think those networks can work with variable inputs, they use fixed size images, svg describes paths that highly vary.
  - abrichr 7y ago
    That's true for convolutional networks. However, recurrent architectures aren't similarly limited, e.g SketchRNN: https://github.com/tensorflow/magenta/blob/master/magenta/models/sketch_rnn/README.md https://github.com/tensorflow/magenta/blob/master/magenta/mo...

CharlesW 7y ago

I'm a little confused by the assertion of "artificial intelligence's preference for texture over shape". The article makes it sound like an intrinsic issue with AI, when to my understanding it's just an outcome of popular image classification algorithms. Couldn't one create classifiers that depend more heavily on object shapes?

swagasaurus-rex 7y ago
I'd guess animals have a lot of compute dedicated to finding out the shape based off of various visual cues: Lighting changes indicating edges Gradients indicating a smooth curve Texture maps that discern between matte, smooth, translucent, often in depth enough to determine the texture material itself. Reflectivity Changes in reflectivity when reorienting oneself or the object Binocular Vision Parallax effect between frames A large database matching objects to memories and experiences Many other techniques for tracking and seeing movement. In the above, Parallax and Binocular vision, and consulting one's memory are the only ways to determine the sizes of the objects being looked at. It's why those optical illusions tiny/big rooms work so easily; neither we nor CV algorithms would be able to discern the sizes of objects without prior memory of such objects or by walking around it.

the_seraphim 7y ago

My question is how do we then deal with the situation where a group of cats is making up the shape of an elephant? a good AI will eventually need to be able to identify this subtle difference, it is an elephant, made of cats, not a cat and not an elephant.

udayrddy 7y ago

So, a person with a gray texture of elephant skin shirt is classified as elephant ?!

liuliu 7y ago
Yes, maybe. But it is at least unclear whether is inherently CNN or not (many people would vote for no). ImageNet training data is heavily texture dependent, therefore, the trained model leaning towards texture information more. Some recent papers did style transfer to randomize textures in order to train the model to focus more on the shape, the reported results seem good.
- w_t_payne 7y ago
  Exactly. I would also add (although without strong evidence to support my assertion) that it seems to be more a function of the training data than of the network architecture.
Sharlin 7y ago
One of the most archetypal failure modes of current NN-based classifiers is, for example, happily reporting seeing a spotted feline given a picture of a couch with a leopard pattern.
- pmontra 7y ago
  Well, if I'm given the task to enumerate what I see in that image maybe I would also add the feline label. But I would add the picture_of attribute, if that's a thing.
  - Sharlin 7y ago
    I don’t think today’s classifiers are anywhere sophisticated enough to have a concept of something vs. a picture of something, a type of use–mention distinction, or ceci n’est un pipe if you will. The point was that people think they trained their classifier to recognize leopards, but in actuality it just learned to recognize the leopard coat pattern.

ryu2k2 7y ago

Just a temporary restriction until we start training AI in robots by interaction with the real world.

w_t_payne 7y ago

Depends on the training data. These networks are correlation-finding machines, and will simply latch onto the simplest correlation possible that produces the correct results. Only by explicitly controlling the correlations in the training data can you force the network to focus on the attributes that you wish. This is why domain-randomization (such as the approach that NVIDIA is taking) helps support generalization -- it's a very direct and efficient form of regularization, removing correlations that do not generalize, forcing the network to work harder to find correlations that do generalize. This also has the added benefit of being something that we can tie back to requirements and physical properties, an important ingredient in making these systems understood and safe.

jcranendonk 7y ago

My take on this: if we want vision ML to succeed at recognition in the same way as humans, perhaps we need to pre-process and present visual information in the same way as the human vision system? As far as I'm aware, we get a lot of info from our eyes about lines and orientation that assists in recognizing shapes. I'm not well-informed about the current state of visual recognition DL, perhaps someone who is can tell us more about whether that approach makes sense.

tveita 7y ago
When you train a deep convolutional neural network, the first couple of layers appear to take on this role, detecting simple features like edges and textures, which the higher layers build upon to see more complex objects. For example https://www.researchgate.net/figure/Visualization-of-example-features-of-eight-layers-of-a-deep-convolutional-neural_fig2_279068412 https://www.researchgate.net/figure/Visualization-of-example..., where you can see (somewhat, if you zoom in) that layer 1 neurons are interested in very simple features, like strong horizontal edges, or particular gradients.

quico 7y ago

Yann LeCun (Chief AI @ FB) puts this article in perspective: https://www.facebook.com/yann.lecun/posts/10156068737942143 https://www.facebook.com/yann.lecun/posts/10156068737942143 TLDR: The use of texture is inherent to the ImageNet dataset and not to deep learning / ConvNet. Training on less-textured versions of ImageNet drives the ConvNet to focus more on shape.

hooloovoo_zoo 7y ago
Sure, but just because you can coerce an algorithm into doing something different does not mean the algorithm itself does not have fundamental tendencies.
p0nce 7y ago
Yes but what the NN exchange between layers is fundamentally textures, not shape descriptions right ?

modeless 7y ago

There is a misunderstanding here. It's not the algorithm that emphasizes textures over shapes, it's the dataset. Texture is a more informative feature than shape in the datasets used. "AI" can detect shapes just fine; simply use a different dataset and you'll get a different result with the same algorithm. I mean, just look at the classic MNIST: it's all shape, no texture, and neural nets work great.

Majromax 7y ago
> Texture is a more informative feature than shape in the datasets used. But the insight is that although both features are informative, classical training approaches result in networks that over-fit on texture information and ignore shape information. Table 2 of the discussed paper finds that joint training with "Stylized ImageNet" (IN images after a style transfer) and classic IN followed by a fine-tuning pass on IN results in improved classification accuracy over IN training alone.
- skgoa 7y ago
  Which proves exactly what the top level comment claimed: the distribution of examples in the dataset was incorrect for the learning task.
  - yarg 7y ago
    Not necessarily. Textures are high frequency low abstraction data forms, shapes are low frequency high abstraction data forms. The former are easier to learn for multiple reasons: The nature of higher frequency data means that you have more samples. (There are far more furry textures in a picture of a cat than there are toes, paws, legs etc.) Low abstraction data forms are also learnt in earlier layers of the network - more abstract forms are resolved through the understanding of how less abstract forms group together, making the low abstraction forms a prerequisite to more in depth learning.
anbop 7y ago
One thing that’s interesting about textures is that humans have an incredible sensitivity to it by touch. You can detect the difference between a flat surface and a surface with angstrom-magnitude features.

stcredzero 7y ago

I can't wait for the Hollywood version of this idea, à la the Infrared viewer from Predator or the machine's eye view from Terminator.

jpalomaki 7y ago

Babies use their hands to feel shapes. Maybe brain works hard to make the connection between the inputs provided through touch and vision. Understanding shapes/outlines is useful for manipulating objects and avoiding obstacles. Maybe the object recognition develops a bit later and builds on top of how brains already understand the world.

scotty79 7y ago
I think babies also have pretty crappy sharpness of vision. Maybe that's on purpose to help with initial learning how to see...
- mkl 7y ago
  It seems they reach healthy adult sharpness by 6 months [1]. By the time they are sitting and starting to move they are certainly extremely good at finding tiny bits of fluff etc. to put in their mouths! [1] https://en.wikipedia.org/wiki/Infant_visual_development https://en.wikipedia.org/wiki/Infant_visual_development

Havoc 7y ago

I thought that's why edge detection preprocessing etc is in use? Doesn't quite seem like a break through insight to me?

jonplackett 7y ago

Does this have anything to do with AIs always learning from 2D photos when we learn from binocular vision so every image contains a lot more ‘shape’ information.

esotericn 7y ago
It seems like that's fairly important to me. Consider that even if you're monocular, you're still able to determine parallax by moving your head around to determine depth.
anewhnaccount2 7y ago
Not only this, but we when we see things even when we are still our eyes/head move a small amount. Shape information is preserved in these multiple perspectives relatively more than texture information.