I'd like to add something to the discussion wrt network architectures and representations of neural net parameters. Neural nets when being trained can be packed

I'd like to add something to the discussion wrt network architectures and representations of neural net parameters. Neural nets when being trained can be packed up as one parameter vector to represent this same structure. In fact, this is how a lot of linear optimizers train neural networks. To demonstrate a quick example: A typical neural net layer is made up of a weight matrix W and a bias vector. This represents the connections of the network. A typical feed forward architecture will have a weight matrix of number of inputs x number of outputs with a bias equal to the number of outputs. Say: W is a 3 x 2 with a bias of 2. We can then represent a neural net parameter space as an 8 length vector. The only the the neural net needs to know how to do is "unpack" the parameters to retain the layer structure. I actually do this in deeplearning4j for training the weights and putting them in to a search algorithm. You would then repeat this for however many layers you have in your network going in order to the output layer. Anyways: the reason deep architectures are used with more "learning" capacity is due to being able to intermix different kinds of activations. A typical example of this is a deep belief network where you have an initial layer that takes in continuous data and then the activations change it to binary. It really depends on the problem you're solving as to whether this is relevant or not. Just a neat thought. Anyways great paper!