Regardless of whether this post is right or wrong, it is a showcase for the kind of out-of-the-box thinking that can force people within and from outside the fi

Regardless of whether this post is right or wrong, it is a showcase for the kind of out-of-the-box thinking that can force people within and from outside the field to see deep neural nets from a different angle. For that alone, I think this is a fantastic essay. Some comments: The author writes: "using multiple copies of a neuron in different places is the neural network equivalent of using functions. Because there is less to learn, the model learns more quickly and learns a better model. This technique – the technical name for it is 'weight tying' – is essential to the phenomenal results we’ve recently seen from deep learning. Of course, one can't just arbitrarily put copies of neurons all over the place. For the model to work, you need to do it in a principled way, exploiting some structure in your data. In practice, there are a handful of patterns that are widely used, such as recurrent layers and convolutional layers." Recurrent and convolutional layers are not just two examples of widely used "weight tying" patterns; they are THE TWO MAJOR WAYS in which virtually everyone "ties" weights -- across time or pixel space, usually. With the exception of tiny fully-connected models used to solve relatively small problems, every successful deep neural net model of meaningful scale I know of is either a convnet or an RNN! If anyone knows of a counter example, I'd love to hear about it! I would also have pointed out that "weight tying" not only allows the deep neural net to learn more quickly; it also massively reduces the "true" number of parameters required to specify the model, thereby reducing by a similar proportion the number of samples necessary to have reasonable PAC learning guarantees.