3 ms·

Have you read Rich Cuarana's work on approximating the behavior of deep nets with single hidden layer networks? http://arxiv.org/pdf/1312.6184.pdf http://arxiv

Have you read Rich Cuarana's work on approximating the behavior of deep nets with single hidden layer networks? http://arxiv.org/pdf/1312.6184.pdf http://arxiv.org/pdf/1312.6184.pdf From that paper, it seems like it's just that deep nets are easier to train to find state of the art solutions than shallow nets. It's not that shallow nets are inherently incapable of performing as well. Shallow nets can mimic/approximate a given well trained deep net and preserve almost all of the accuracy. So it's not the case that a good solution to the task doesn't exist in the set of hypothesizes spanned by sized shallow net, it's just that people don't know how to effectively find the right parameters for the shallow nets.

srean 12y ago
Minor comment, please link the abstract and not the pdf. Those who want to follow up can pull the link to the pdf from the arxiv abstract.
kastnerkyle 12y ago
So far this is only shown on TIMIT and MNIST, which are pretty trivial datasets, so it may be dataset dependent. One of the authors is giving a talk at INRIA in France this month, and the abstract mentioned CIFAR10 (possibly unpublished results). If they have manged to compress a CIFAR10 network, that is a strong indicator that ImageNet networks could be compressed in the same way... but no one has published any results in this regard to my knowledge. However this is an active area of research for me, and I hope to explore it more soon. I think there are better ways to approximate than this, but this work at least shows that it may be possible.