One thing to consider is the distinction between deep neural networks as mathematical objects and machine learning as currently practiced. Lately, there have b

One thing to consider is the distinction between deep neural networks as mathematical objects and machine learning as currently practiced. Lately, there have been quite a few theories on neural networks as ideal nonlinear approximators [1]. Similarly, people have shown many ways that gradient descent can tend to reach a global maximum of regularized curve-closeness[2]. Which is to say, if your development cycle is: gather-data, train, test, deploy, we know this approximates the data almost ideally; you can't really do much better than a deep network. But we know in practice, when deployed, that deep neural networks actually have many limitations (compared to our intuitions or compared to human performance, etc). There are some obvious explanations. Of course, they're limited by our ability to gather data and by the biases of the data. But even more, they're limited to situations where you have large chunks of unchanging data that you can extrapolate from. Given that deeps are more or less perfect for the train-test-deploy cycle, it seems like the problem is with this cycle itself. And it's easy to see human beings somehow acting "intelligently" without using this cycle. So figuring out an alternative to this might be something to look at. [1] For example: Nonlinear Approximation and (Deep) ReLU Networks I. Daubechies, et al https://arxiv.org/abs/1905.02199 https://arxiv.org/abs/1905.02199 [2] For example: Gradient descent optimizes over-parameterized deep ReLU networks Difan Zou et al https://link.springer.com/article/10.1007/s10994-019-05839-6 https://link.springer.com/article/10.1007/s10994-019-05839-6