Supervised learning is a classic inverse problem -- you know the desired outcome, but not how to get there. (think control systems) Suppose you had a flexible

Supervised learning is a classic inverse problem -- you know the desired outcome, but not how to get there. (think control systems) Suppose you had a flexible model in whose parameter space you could search. A naive algorithmic approach would be to treat this as a brute force search problem. Maybe you can be slightly more clever algorithmically (better search algorithms, heuristics, etc) But can you somehow be really clever and exploit the fact that the parameter space is continuous, with a smooth cost function? That's what gradient descent basically is. Lots of bells and whistles on top, to exploit extra problem structure. Now, let's go back to our family of models. They just need to be transparent to gradient information -- then you could use simple ideas at the heart of adaptive control theory. Suddenly, the fact that autodifferentiation techniques can differentiate arbitrary computational algorithms (sibling comment explains how compositionality helps this) sounds mouth-watering! That's what makes a lot of people excited -- your model family just expanded to programs performing arbitrary computations on continuous parameters (including loops!) -- not just some silly function approximator like a natural network (exaggerating to make my point)