4 ms·
In what sense are neural nets not rigorous? The whole pipeline from Pytorch or TensorFlow and Python to LLVM, to GPUs or TPUs is absolutely rigorous. Much more
by GregarianChild 6y ago
In what sense are neural nets not rigorous?
The whole pipeline from Pytorch or TensorFlow and Python to LLVM, to
GPUs or TPUs is absolutely rigorous. Much more rigorous,
in fact, than normal, hand-written mathematics, as you find it in
e.g. a typical Annals of Mathematics publication, or mathematical textbook!
I think what you really have in mind is a simple model of modern deep
learning that is not fully accurate, but still useful!
Let me argue by analogy. You are looking for something
that is to deep learning what the lambda-calculus is to the Haskell
compiler. One of the main simplifications in
programming language theory is replacing finite precision arithmetic
(which is painfully complex) with mathematical integers and real
numbers (which are much simpler). Would a theory
of deep learning based on mathematical reals be
valuable in a theory of deep learning? The stunning success of
floating point formats like bfloat16 [1] suggests otherwise, since
arithmetic precision in deep learning is closely connected to important learning
phenomena such as overfitting and regularisation.
I am tempted to be provocative and say that you are really looking for less rigour!
[1] https://en.wikipedia.org/wiki/Bfloat16_floating-point_format https://en.wikipedia.org/wiki/Bfloat16_floating-point_format
- avrionov 6y agoNot the OP, but I've built several ML pipelines in the last few years. Even if every step in the process very rigorous and using solid software, there are still challenges. ML models are very difficult (or impossible to test). The usual testing approaches don't work at all for ML features.
- b3kart 6y agoWe don't know which model architecture will work for which problem, and why it would/wouldn't. We experiment until we find something that works, and can sometimes try to guess why it did. But none of this knowledge is formalized in a way that can reliably predict performance in future problems. We engineer solutions to problems, but don't build a rigorous body of knowledge to help us in future problems.
- glial 6y agoWell put. I would say that often even the problems aren’t particularly well defined.
- GregarianChild 6y agoThe problem you describe (model architecture will work for which problem) is not lacking rigour, but due to the Turing complete expressive power of neural networks. What you can do is use the existing rigour and derive stronger properties for restricted classes of NNs (just like you can prove stronger properties for simple subsystems of lambda-calculus) and that is an interesting field of study.
- eli_gottlieb 6y ago> The problem you describe (model architecture will work for which problem) is not lacking rigour, but due to the Turing complete expressive power of neural networks. Non-recurrent neural networks are not Turing-complete in any sense.
- deleted 6y ago[deleted]
- karpierz 6y ago> In what sense are neural nets not rigorous? In the sense that for non-trivial applications, we struggle to define what approach is successful and why it's successful without leaning on empirical metrics.
- GregarianChild 6y agoThat's not because of lacking rigour, but because of complexity! You cannot, in general, predict arbitrary program properties without running programs, that's the essence of Rice's theorem [1]. By the famous Universal Approximation Theorems for neural networks, in general, NNs are Turing complete, which precludes general and simple mathematical "silver bullets" that will help you overcome those challenges. [1] https://en.wikipedia.org/wiki/Rice%27s_theorem https://en.wikipedia.org/wiki/Rice%27s_theorem
- exporectomy 6y agoI think you're confusing rigorous processes with rigorous theory. The analogy of steam engines seems appropriate - they were initially built with (sort of) rigorous processes for metal forming and joining, etc. It wasn't just people tinkering in their backyards. Yet that was done without the theory of thermodynamics. Those rigorous engineers didn't know a Helmholtz free energy from a free energy machine. Once we had thermodynamics, it didn't directly influence how to form metal parts or lubricate pistons, but it did put hard constraints on the possible performance of engines and made it easier to predict how an engine would perform before building it, even if it was outside the range of what had already been tested - eg. jet and rocket engines that required high confidence in predictions of the theory to invest the money needed to develop working models in the middle of wartime scarcity.