From reading the page and these comments... A feedforward neural network can be viewed as a "continuous generalization" of a fixed-size circuit, which makes it

From reading the page and these comments... A feedforward neural network can be viewed as a "continuous generalization" of a fixed-size circuit, which makes it able to learn any function within the circuit complexity of the network. A deep neural network biases towards circuits that are deeper than they are wide, towards functions composed out of smaller functions. Since many useful functions are such (and for other theoretical reasons, such as deep networks having "funnels" in their free-energy landscapes), deep learning works well. A recurrent neural network goes from circuits to deterministic finite-state automata. LSTM basically turns the whole thing into a small, fixed-size almost sorta-kinda pushdown automaton, and Neural Turing Machines generalize to a sorta-kinda linear-bounded automaton. This stack approach makes things much more like a real pushdown automaton. So a feedforward network with a stack should be able to "learn any PDA", and a recurrent network with an unlimited-size stack should be able to learn more-or-less any Turing machine (though it's biased towards Turing machines that "look like" PDAs with funny state transitions). Kinda. waves hands