Machine learning, specifically deep neural networks, are missing something rather significant as a science: some sort of universal, underlying, fundamental theo

Machine learning, specifically deep neural networks, are missing something rather significant as a science: some sort of universal, underlying, fundamental theorem that explains neural network topographies. Physics have laws of motion, for example. Neural nets are still "twiddle these values and see what happens" -- that kind of science represents more than a small number of papers. This paper suggests that there is a close relationship ("one-to-one mapping") between neural networks and something in physics called variational renormalization groups. They say: > RG plays a central role in our modern understanding of statistical physics and quantum field theory. A central finding of RG is that the long distance physics of many disparate physical systems are dominated by the same long distance fixed points. This gives rise to the idea of universality – many microscopically dissimilar systems exhibit macroscopically similar properties at long distances Physicists have developed elaborate technical machinery for exploiting fixed points and universality to identify the salient long distance features of physics systems. It will be interesting to see, what, if any of this more complex machinery can be imported to deep learning. If we can grasp some theory underpinning neural networks, the science of machine learning will be less "let's try this topography/architecture and see what happens" and will be more of "if we add a layer of this size in this part of the network, we will expect these results, or results to change in <specific way>". Anyway, maybe physics can teach us something about neural networks.