I think another interesting analogy (in terms of field-development trajectory) to deep learning is the whole idea of l1-regularized regression or LASSO or spars

I think another interesting analogy (in terms of field-development trajectory) to deep learning is the whole idea of l1-regularized regression or LASSO or sparsity priors or sparse signal processing ([1] goes by many names depending on which field you work in). The whole idea is that by penalizing 'dense' solutions to a regression problem, you can 'promote' 'sparse' solutions like the ones that occur in many many applications. This had been used by various communities with some theoretical justification for years, at least since the 1970s. However, the real theoretical breakthroughs framing this problem in something close to the rigor and usefulness of Shannon's sampling theorem, came in 2004 from two papers independently: David Donoho; and Emmanuel Candes, Justin Romberg, Terence Tao. Theoretical CS community also got close to the answer in their work in the late 1990s and early 2000s in many papers on random projections and sketching. There are many connections between deep learning and ideas in the broad area of sparse regression. But one epistemological point of intersection is how theoretical results pertain to asymptotically large cases, but in applications work very well even with much smaller systems. [1] https://en.wikipedia.org/wiki/Compressed_sensing https://en.wikipedia.org/wiki/Compressed_sensing