Well, the theory around neural nets strongly suggests that enough nonlinear activation functions combined in the right way should be able to learn any function,

Well, the theory around neural nets strongly suggests that enough nonlinear activation functions combined in the right way should be able to learn any function, including basic arithmetic. Now, whether or not you have the right approach to training the network to get the right set of weights is a different story...