6 ms·

It seems like, to be really good the AI needs to construct an internal 3D model of objects so it won't matter which way it's rotated. It seems to be how the hu

It seems like, to be really good the AI needs to construct an internal 3D model of objects so it won't matter which way it's rotated. It seems to be how the human works. I can rotate an object in my mind and picture it from any angle.

StavrosK 7y ago
You can, but I don't think that's what happens when looking at stuff. When I look at the hammer, I don't rotate a hammer to see what orientation matches what I'm seeing, I see the handle and... metal bit (don't know the term, sorry), realize those look like they belong to a hammer and go "oh yeah, it's an upside down hammer".
- tcgv 7y ago
  It took me a while to recognize that wooden chair. At first I tought it was a wooden hammer on top of some squared background, but then I realized it was a chair seen from above. For me, I actually imagine these objects moving/rotating to make sense of them when seen from unusual angles. That hammer you described, I look at it and imagine myself flexing it.
  - bonoboTP 7y ago
    Another thing that seems overlooked is that we don't just randomly happen to look at objects from weird angles. If you're looking at a chair from that weird top-down angle, probably you walked to it previously and expect to see it like that. And you feel the direction of gravity in your ear, so you feel your viewpoint. The object itself may still be in a random orientation, but then just moving your head around a bit or turning the object clears up any confusion. We do heavily rely on context.
  - pauljurczak 7y ago
    Same here. I needed almost 10 seconds to recognize the chair.
musingsole 7y ago
For contrast, horse brains seem to not do this. They'll often see the same object from a different angle as a new stimulus.
IshKebab 7y ago
I think that's the idea behind capsule networks. But I haven't heard anything more about them since they were announced so maybe they don't work too well.
bonoboTP 7y ago
Decades ago, computer vision was primarily thought of as a kind of inverse graphics, the opposite of rendering: image goes in, 3D shapes with material properties and lighting properties go out. Of course 3D reconstruction is still a huge thing, but object recognition split off onto a different path with the realization that "superficial", 2D-based features (like SIFT and HOG) work very well for recognizing image content, when combined with powerful classifiers and regressors of the time (like the SVM). It was common for lecturers to say that "You may think we need complicated internal 3D representations of everything, but another approach seems more fruitful: ..." Nowadays there's a lot of buzz around merging the two branches back together to unify explicit 3D geometric reasoning and 3D modeling with deep learning. I think humans use both strategies. Sometimes we really rely on superficial visual information, like yellow/black stripes -> time to get away! No need to first perfectly match the visual input to a mental tiger model rotated at the correct orientation. I think split-second recognition is usually like this. Or perhaps we use different strategies for different objects, I could imagine for example that facial recognition in the brain is more 2D-feature based pattern matching, rather than 3D reconstruction.
AareyBaba 7y ago
"I can rotate an object in my mind and picture it from any angle" Except there was a wonderful thread about Aphantasia a while ago https://news.ycombinator.com/item?id=20267445 https://news.ycombinator.com/item?id=20267445 Several HN readers chimed in to say they have this feature. It would be interesting to know if this dataset would stump them.
JMTQp8lwXL 7y ago
The model hasn't ever picked up or held a hammer. How is it ever supposed to recognize a hammer in an unfamiliar context (e.g., when given a new image that is not representative of what it was trained against?). You can't train intuition.
rstuart4133 7y ago
> It seems to be how the human works. I can rotate an object in my mind and picture it from any angle. You can, but that doesn't seem to me to be how the mind works. If it looked at it from every angle simultaneously there would be no speed difference in my recognising an object regardless of the orientation. But in some cases there are a huge speed difference. I can be staring at it for many seconds and then snap! - oh it's upside down. As soon as I realise this my mind immediately adapts and what was unrecognisable is suddenly as plain as day. The only way I can account for this is when I finally twig the image is upside down I re-route it through a different path in my brain that does a rotation before feeding it to the recogniser. But normally that path is shut off - it's not constantly scanning the input. I suppose what happens is in most cases some pre-processor uses other clues in the picture to tell me it's rotated from it's normal position and engages the correct path without conscious intervention. That would explain why most of the time you say you don't notice it. Nonetheless the two mechanisms are very different. A sequential path that rotation -> recognition will be slightly deeper and slightly slower than that does both in the one step, but far smaller. Nonetheless, to looks to me modern designs do attempt to do it in one step, which is to say they attempt to recognise the object in all possible orientations simultaneously.

StavrosK 7y ago

You can, but I don't think that's what happens when looking at stuff. When I look at the hammer, I don't rotate a hammer to see what orientation matches what I'm seeing, I see the handle and... metal bit (don't know the term, sorry), realize those look like they belong to a hammer and go "oh yeah, it's an upside down hammer".

tcgv 7y ago
It took me a while to recognize that wooden chair. At first I tought it was a wooden hammer on top of some squared background, but then I realized it was a chair seen from above. For me, I actually imagine these objects moving/rotating to make sense of them when seen from unusual angles. That hammer you described, I look at it and imagine myself flexing it.
- bonoboTP 7y ago
  Another thing that seems overlooked is that we don't just randomly happen to look at objects from weird angles. If you're looking at a chair from that weird top-down angle, probably you walked to it previously and expect to see it like that. And you feel the direction of gravity in your ear, so you feel your viewpoint. The object itself may still be in a random orientation, but then just moving your head around a bit or turning the object clears up any confusion. We do heavily rely on context.
- pauljurczak 7y ago
  Same here. I needed almost 10 seconds to recognize the chair.

musingsole 7y ago

For contrast, horse brains seem to not do this. They'll often see the same object from a different angle as a new stimulus.

IshKebab 7y ago

I think that's the idea behind capsule networks. But I haven't heard anything more about them since they were announced so maybe they don't work too well.

bonoboTP 7y ago

Decades ago, computer vision was primarily thought of as a kind of inverse graphics, the opposite of rendering: image goes in, 3D shapes with material properties and lighting properties go out. Of course 3D reconstruction is still a huge thing, but object recognition split off onto a different path with the realization that "superficial", 2D-based features (like SIFT and HOG) work very well for recognizing image content, when combined with powerful classifiers and regressors of the time (like the SVM). It was common for lecturers to say that "You may think we need complicated internal 3D representations of everything, but another approach seems more fruitful: ..." Nowadays there's a lot of buzz around merging the two branches back together to unify explicit 3D geometric reasoning and 3D modeling with deep learning. I think humans use both strategies. Sometimes we really rely on superficial visual information, like yellow/black stripes -> time to get away! No need to first perfectly match the visual input to a mental tiger model rotated at the correct orientation. I think split-second recognition is usually like this. Or perhaps we use different strategies for different objects, I could imagine for example that facial recognition in the brain is more 2D-feature based pattern matching, rather than 3D reconstruction.

AareyBaba 7y ago

"I can rotate an object in my mind and picture it from any angle" Except there was a wonderful thread about Aphantasia a while ago https://news.ycombinator.com/item?id=20267445 https://news.ycombinator.com/item?id=20267445 Several HN readers chimed in to say they have this feature. It would be interesting to know if this dataset would stump them.

JMTQp8lwXL 7y ago

The model hasn't ever picked up or held a hammer. How is it ever supposed to recognize a hammer in an unfamiliar context (e.g., when given a new image that is not representative of what it was trained against?). You can't train intuition.

rstuart4133 7y ago

> It seems to be how the human works. I can rotate an object in my mind and picture it from any angle. You can, but that doesn't seem to me to be how the mind works. If it looked at it from every angle simultaneously there would be no speed difference in my recognising an object regardless of the orientation. But in some cases there are a huge speed difference. I can be staring at it for many seconds and then snap! - oh it's upside down. As soon as I realise this my mind immediately adapts and what was unrecognisable is suddenly as plain as day. The only way I can account for this is when I finally twig the image is upside down I re-route it through a different path in my brain that does a rotation before feeding it to the recogniser. But normally that path is shut off - it's not constantly scanning the input. I suppose what happens is in most cases some pre-processor uses other clues in the picture to tell me it's rotated from it's normal position and engages the correct path without conscious intervention. That would explain why most of the time you say you don't notice it. Nonetheless the two mechanisms are very different. A sequential path that rotation -> recognition will be slightly deeper and slightly slower than that does both in the one step, but far smaller. Nonetheless, to looks to me modern designs do attempt to do it in one step, which is to say they attempt to recognise the object in all possible orientations simultaneously.