3 ms·

Time shift invariance, which is used when you assume that past patterns predict future results, is one form of invariance. Space shift invariance and rotation

Time shift invariance, which is used when you assume that past patterns predict future results, is one form of invariance. Space shift invariance and rotation invariance are others. You have to specifically program your nn to look for time shift invariance; stands to reason you'd want to design your architecture for the spacial invariances too. In other words: it's not reasonable to expect a neural net to derive space and rotation invariance from first principles, and I'd be very surprised to learn that the human brain didn't have special purpose hardware for accomplishing those things.

dr_zoidberg 8y ago
You made me think of the Margaret Thatcher Illusion, for which the best example I found was Dr. Phil Plait[0]. Funny thing, the idea of Capsule Networks (Hinton et al)[1, 2] seems to tackle some of these issues, though it's they're a bit young yet and more more work. [0] https://www.opticalspy.com/opticals/dr-phil-plait-optical-illusion https://www.opticalspy.com/opticals/dr-phil-plait-optical-il... [1] https://arxiv.org/abs/1710.09829 https://arxiv.org/abs/1710.09829 [2] https://hackernoon.com/what-is-a-capsnet-or-capsule-network-2bfbe48769cc https://hackernoon.com/what-is-a-capsnet-or-capsule-network-...
- fallingfrog 8y ago
  I think that convolutional networks are really mostly a way to address space shift invariance too; and I even wonder if the same job could be done by some special purpose code that just runs the underlying neural net with a bunch of different rotations of the same image. That's probably how they do it now.. I feel like that's probably close to the optimal approach.
magicalhippo 8y ago
Having dabbled with image processing, your comment reminded me of the Fourier Mellin transform, which can be used for translation, rotation and scale invariant feature detection. I did a quick search and came up with this paper[1], and while they don't use the Fourier-Mellin transform, they do use a log-polar transform[2], where rotation and scaling are transformed to translations. This, they claim, result in much improved classification of rotated and scaled data. I haven't been keeping up on the latest in the ML field tho, so maybe it's rubbish, but like you I'd be surprised if some extra processing doesn't play a role. [1]: https://arxiv.org/abs/1709.01889 https://arxiv.org/abs/1709.01889 [2]: https://sthoduka.github.io/imreg_fmt/docs/log-polar-transform/ https://sthoduka.github.io/imreg_fmt/docs/log-polar-transfor...
candiodari 8y ago
> the human brain didn't have special purpose hardware for accomplishing those things. How about a head mounted on a ball joint ? Would that count ? (since obviously it would introduce small spatial variations VERY often). You can make a similar argument for being mounted on a mobile robot (like the human or any other animal body).

dr_zoidberg 8y ago

You made me think of the Margaret Thatcher Illusion, for which the best example I found was Dr. Phil Plait[0]. Funny thing, the idea of Capsule Networks (Hinton et al)[1, 2] seems to tackle some of these issues, though it's they're a bit young yet and more more work. [0] https://www.opticalspy.com/opticals/dr-phil-plait-optical-illusion https://www.opticalspy.com/opticals/dr-phil-plait-optical-il... [1] https://arxiv.org/abs/1710.09829 https://arxiv.org/abs/1710.09829 [2] https://hackernoon.com/what-is-a-capsnet-or-capsule-network-2bfbe48769cc https://hackernoon.com/what-is-a-capsnet-or-capsule-network-...

fallingfrog 8y ago
I think that convolutional networks are really mostly a way to address space shift invariance too; and I even wonder if the same job could be done by some special purpose code that just runs the underlying neural net with a bunch of different rotations of the same image. That's probably how they do it now.. I feel like that's probably close to the optimal approach.

magicalhippo 8y ago

Having dabbled with image processing, your comment reminded me of the Fourier Mellin transform, which can be used for translation, rotation and scale invariant feature detection. I did a quick search and came up with this paper[1], and while they don't use the Fourier-Mellin transform, they do use a log-polar transform[2], where rotation and scaling are transformed to translations. This, they claim, result in much improved classification of rotated and scaled data. I haven't been keeping up on the latest in the ML field tho, so maybe it's rubbish, but like you I'd be surprised if some extra processing doesn't play a role. [1]: https://arxiv.org/abs/1709.01889 https://arxiv.org/abs/1709.01889 [2]: https://sthoduka.github.io/imreg_fmt/docs/log-polar-transform/ https://sthoduka.github.io/imreg_fmt/docs/log-polar-transfor...

candiodari 8y ago

> the human brain didn't have special purpose hardware for accomplishing those things. How about a head mounted on a ball joint ? Would that count ? (since obviously it would introduce small spatial variations VERY often). You can make a similar argument for being mounted on a mobile robot (like the human or any other animal body).