4 ms·
Here's a problem: you can argue that Gabor filters arise because we design neural nets to encourage them. Gabor filters mostly arise in CNN or things otherwise
by argonaut 10y ago
Here's a problem: you can argue that Gabor filters arise because we design neural nets to encourage them. Gabor filters mostly arise in CNN or things otherwise regularized to be like CNNs. Convolutional layers are a form of regularization that restrict the space of models that a network can conform to. The Gabor filters are learned but none of this is evidence they are globally "optimal" given that a human manually decided whether or not to include the presence of convolutions.
It also goes without saying that the phrase "statistically optimal" is meaningless in this specific context. You can claim they are a part of minimizing the cost function, but, again, you have to be very careful about the chicken and egg problem, because humans are the ones who manually craft the cost function.
- suchow 10y agoYou might find this 1996 Nature paper by Olshausen & Field interesting. In it, they describe how a coding strategy that maximizes spareness when representing natural scenes is enough to produce a family of localized, oriented, bandpass receptive fields, like those found in the early visual system of humans. https://courses.cs.washington.edu/courses/cse528/11sp/Olshausen-nature-paper.pdf https://courses.cs.washington.edu/courses/cse528/11sp/Olshau...
- happycube 10y agoSuperficially, those graphics look a lot like visualizations of conv nets.
- argonaut 10y agoThis fits in with my point: they imposed several restrictions about the model space: maximizing sparseness, and they also make several linearity assumptions.
- eli_gottlieb 10y agoGreat, now I've got another paper to cite. Thanks!
- trhway 10y agothose papers should be, if not 101, at least 201 for neural net track as it would help to establish common framework and basis for thinking, talking and analysis of neural nets machinery.
- deleted 10y ago[deleted]
- hacker42 10y agoI am not seeing the chicken and egg problem. Isn't it always the case that when we consider the optimality of something it depends on some definitions?
- argonaut 10y agoThe chicken and egg problem is that we design neural nets in order for Gabor filters to show up. If you used a different neural net architecture choice, they wouldn't show up. So the presence of Gabor filters indicating optimality is sort of begging the question.
- trhway 10y agoCNN beat other architectures in images and Gabor emerges in CNN. That is a pretty good candidate for even global optimaliy.
- argonaut 10y agoThe fact that X works and A, B, and C don't work, does not imply A is globally optimal.
- hacker42 10y agoAs far as I know, Gabor filters show up in all kinds of multi-layered models, even in unsupervised ones. Gabor filters are quite canonical.
- trhway 10y ago>you can argue that Gabor filters arise because we design neural nets to encourage them. Gabor filters mostly arise in CNN or things otherwise regularized to be like CNNs. and do we know why? Usually it would mean some optimality. It should be relatively simple math here (back at the time at our University it would be given to a student as a thesis project and couple months later we'd have it), and that would give us 2 things - insight into biological visual cortex (which we suppose follows some optimality too and know we would have a very good candidate for the one) as well as to allow to start some primary convolutional layers with the (optimal set of) Gabors instead of going through learning them. Actually some of the best results i saw 15-20 years ago were produced by the simulation of visual cortex through such construction. And now image trained deep learning nets converge to the same.
- argonaut 10y agoNo, we don't know why. You also have little basis to claim the biological visual cortex is optimal. It certainly works, which is enough for people who draw inspiration from it.
- trhway 10y agothat is my point - we have very good suspicion and experimental data (successful deep learning CNN as well as visual cortex) that it is optimal, at least in some very wide class if not global, and that warrants investigation for a proof of it. Proven optimality would be very telling, especially for biological visual cortex. Even if optimality doesn't happen for CNN Gabor, there may happen to be discovered reasons for why not, and thus that would help to construct even better, probably optimal, approach.
- argonaut 10y agoSuspicion and empirical evidence cannot prove something is optimal. You cannot exhaustively empirically search an infinite space of models. You are seriously misunderstanding the definition of "optimal."