3 ms·
Hmm... it could very well be true. Perhaps deep networks work well because they learn to memorize "the most common patterns of auto-correlation" they see in th
by cs702 10y ago
Hmm... it could very well be true.
Perhaps deep networks work well because they learn to memorize "the most common patterns of auto-correlation" they see in the training data at different levels of function composition.
In fact, we do this explicitly in convolutional layers, which by design learn to represent every input sample as a combination of a finite number of fixed-size square filters.
...and the reason why deep networks might be "generalizing" so well is because not all distributions of natural data are equally likely!
In practice, objects with the same or similar labels tend to lie on or close to lower-dimensional manifolds embedded in data space.
...and this concentration of natural data distributions might be a result of the laws of Physics of the universe in which we happen to live: https://arxiv.org/abs/1608.08225 https://arxiv.org/abs/1608.08225
So, yes, deep learning could very well be a really fancy form of memorization.
I hadn't thought of it in this way before. Very interesting :-)
- 21 10y agoHumans also memorize - what a cat is, what a dog is, a giraffe. Sure, we can apply even higher level features, and we can generalize from a picture of a cat to a black&white or text description of it. But we still memorize a lot.
- yters 9y agoAnimals memorize too. But, humans generalize, and that's the secret to our success. That's why generalization is a big deal and not memorization.
- yters 10y agoAre NNs any better than a simple nearest neighbor algorithm then? It's hard for me to understand the hype if deep learning is just fancy memorization.
- cs702 10y agoYes, deep neural nets are better at many AI/cognitive tasks, as they learn to recognize (and perhaps only memorize) patterns in the data at multiple levels of function composition -- that is, at multiple levels of abstraction. That last bit about "multiple levels" is key. Shallow models like kNN, SVMs, Gaussian Processes, etc. don't do that; they learn to recognize/memorize patterns at only one level.
- yters 9y agoSeems we could just layer any of these other techniques then. The big thing is layers, and neural networks just get traction because people think we're discovering something important about the brain and mind. So it's just PR at the end of the day. Not a true breakthrough.
- cs702 9y agoFirst, yes, that's pretty much what deep neural networks are: layers of shallow machine models stacked atop each other, with each layer learning to recognize patterns at different scale; and we train all layers together end-to-end. Second, it's not PR! This stacking of layers, when done right, can overcome the "curse of dimensionality." Shallow models like kNN, SVM, GP, etc. cannot overcome it; they perform poorly as the number of input features increases. For example, k-nearest-neighbors will not work with images that have millions of pixels each. Third, I'm only scratching the surface here. There's a LOT more to deep learning than just stacking shallow models.