Think about neural nets as emergent: simple nodes give rise to complicated dynamics. The dynamics are determined by the overall geometry of the system. Theore

Think about neural nets as emergent: simple nodes give rise to complicated dynamics. The dynamics are determined by the overall geometry of the system. Theoretically, we should be able to look at a bunch of different geometries and explain what works and what doesn't, just from the original rules and the geometry. The fact that we can't, and usually experimentation is necessary to get a good idea about the strengths and weaknesses of different geometries, is a signal that we don't really understand the system. I'm not aware of any theory of the type: give me an ordered graph with typed nodes and I'll give you the types of data this is good at modeling, how quickly it will converge, if you need dropout or layer-skip or something else in order to make it converge well, etc. In your physics analogy: we can write out various properties of composite particles purely from theory: i.e. Mesons decay pathways and allowed compositions, Baryons decay pathways and allowed compositions, etc, just knowing about the core 17 standard model particles. We know about a few core nodes of neural nets, but can't use that to predict the behavior of the whole net.