A very cynic and provocative follow-up to this hypothesis (if it turns out to be true) would be: DNNs are nothing more than a huge set of NNs where at least on

A very cynic and provocative follow-up to this hypothesis (if it turns out to be true) would be: DNNs are nothing more than a huge set of NNs where at least one happens to solve your problem and the DNN finds out which one. The approach is then essentially the same as a Random Forest with bagging but with the decision trees substituted by Neural Networks. This in turn would mean that: 1. The "Deep <Your Algorithm here>" revolution is in fact not a revolution but just throwing many models (and thereby resources) at the same problem while obscuring that you don't have any idea which one will work because the DNN will do it for you. 2. The age old problem of initializing the neural network is not at all solved and can lead to drastically better results if finally somehow addressed.