3 ms·
Deep learning wasn't novel in 2012 either - it was the removal of a hardware limitation that made it compelling again. I think the same is true for evolving DNN
by frisco 9y ago
Deep learning wasn't novel in 2012 either - it was the removal of a hardware limitation that made it compelling again. I think the same is true for evolving DNNs, but I don't know if the available compute power is there yet.
> My feeling is that since shallow networks can be made to have equivalent accuracy to deep networks, that the real challenge isn't topology but training.
This is not really true though... even very shallow neural networks can be universal function approximators in a trivial sense because they can be lookup tables, but they are really not expressive enough to generalize well and lack a lot of the expressivity of deep networks.
- nurettin 9y agoShallow networks would probably differ in the number of categories they can differentiate with accuracy.
- argonaut 9y ago> shallow neural networks can be universal function approximators in a trivial sense because they can be lookup tables, but they are really not expressive enough to generalize well You've got it flipped. If anything, shallow neural networks (of an equivalent number of parameters) are more "expressive" than deep networks, BUT that expressivity just makes them overfit. This is the bias vs. variance tradeoff. If anything, deep networks encode our prior belief that there is a hierarchy of features / a compressed representation, which limits the model that is learned, to a model conforming to those priors.