Think of it this way: Neural networks are a particular kind of formula. For instance a 2x2 neural network takes 2 inputs, x, and y, has 2 biases, bx and by, and

Think of it this way: Neural networks are a particular kind of formula. For instance a 2x2 neural network takes 2 inputs, x, and y, has 2 biases, bx and by, and 4 weights w1 ... w4 and calculates 2 outputs ox, and oy. The formula it uses is: ox = w1 * x + w3 * y + bx oy = w2 * x + w4 * y + by And then learns how to change w1 ... w4, bx and by to get ox and oy to be more useful (less error). Is it so weird that a 2 layer network (which is the same formula, but it uses ox and oy as inputs to a second identical calculation to produce the final outputs) will produce better results, even with learning ? Even in a neural network so absurdly large as the human mind, it isn't just a bunch of wires connecting everything to everything. There are clear patterns, clear formulas that are dictated not by learning, but by the architecture itself (e.g. CNNs in the eyes and optical cortex, and something much more like LSTMs in memory heavy regions). In some sense one might even say it doesn't make a difference. A fully connected network with the same fanout as a CNN can do everything a CNN can, and more. Likewise, a network that is simply presented with the last 50 timesteps can do strictly better than an LSTM or GRU RNN would. But such a network would have much, much greater computational complexity (by a factor in the thousands at least, in the case of those CNNs by a factor that is roughly the number of pixels in the image). So they don't work better per se. In fact they are "worse" by the most important metric (error). But they are a good trade off : LSTMs are thousands of times faster, with only slightly worse results for sequence labeling or production (ie. text and audio comprehension). CNNs are millions of times faster at image comprehension than a fully connected network, and are only slightly worse at it.