As an expert in both the umbrella of methods known as compressed sensing and more recently deep learning, my opinion is that these techniques are largely comple

As an expert in both the umbrella of methods known as compressed sensing and more recently deep learning, my opinion is that these techniques are largely complementary. Deep learning is fantastic at for instance building appearance models and exploiting hierarchical nature of natural images, and hierarchical feature building in general where appropriate, while methods from compressed sensing remain as quite useful general methods via which one can efficiently extract low dimensional structure, from noisy and at times even highly corrupted datasets. It is true that deep learning is now state of the art in certain tasks such as super-resolution for natural images, which were previously the domain of linear inverse problems, and this is due to its ability to learn useful natural image priors, something that isn't possible with simpler linear or slightly non-linear models. Meanwhile, compressed sensing style methods still excel in situations where the data does not benefit from hierarchical compression, labeled data is not available and/or the data is highly corrupted. Take for example the Netflix challenge problem, discussed in the video, for which deep learning is unlikely to offer substantial benefits, at least for the problem as stated (we just observe partial information about movie ratings). Where deep learning could potentially help in that situation is for instance grouping movies according to high level semantic information derived from text descriptions, other metadata and even the video content of the films themselves, which are still somewhat open problems and would not necessarily add value, depending on the validity of the low rank assumption of movie preferences. More recently studied problems such as phase retrieval, which are in a sense the most elementary non-linear inverse problems, have now been understood, and in fact have informed understanding of how information propagates in deep neural networks (http://yann.lecun.com/exdb/publis/pdf/bruna-icml-14.pdf http://yann.lecun.com/exdb/publis/pdf/bruna-icml-14.pdf). More generally, the study of favorable outcomes in non-convex optimization, which is informed by recent developments in the umbrella field of compressed sensing, will help drive understanding of what makes training deep neural networks possible and thus to improve it, with the current empirical performance of deep learning being far ahead of any theoretical understanding. Broadly speaking, as opposed to fighting about the relevance of one field or the other, we should strive to achieve better overall results by using both sets of techniques complementarily.