These networks ought to be robust to minor changes in W. It's the topology that maters and frankly most of the W_ij != 0 are spurious connections -- meaning per

These networks ought to be robust to minor changes in W. It's the topology that maters and frankly most of the W_ij != 0 are spurious connections -- meaning perturbation analysis will show that they play no causal role in the computation. I wrote a paper on this which has >100 citation (Survival of The Sparsest: Robust Gene Networks are Parsimonious). I used gene networks, but this is just a special case of neural networks. In fact there been a bunch of papers published on gene regulatory networks that show that topology is the main driver of function -- not surprising, if you show the circuit diagram of an 8-bit adder to an EE, they'll know exactly the function. Logically it has to be so. In fact you can model the gene network of the drosophila segmentation pattern with Boolean (1-bit) networks. The problem with ANN research is that no few take the time to understand why things function as they do. We should be reverse engineering these from biology. Every time a major advancement is made in ANNs neurobiologist say "yes, we could have told you that ten years ago" deep learning is just the latest example. It will hit its asymptote soon, then people will say that AI failed to live up to its expectation, then someone will make a new discovery. It's very frustrating to sit on the sidelines and watch this happen again and again.