If you look at the video, I try to give some intuition about this. The following explanation is an oversimplification to give you intuition into what is going

If you look at the video, I try to give some intuition about this. The following explanation is an oversimplification to give you intuition into what is going on: The difficulty in training a deep architecture is that, if you use standard backprop, the gradient signal doesn't flow back to the lowest layers. Instead, the few supervised output units pass back a small gradient signal, and at each layer the gradient signal has increasing noise as it gets passed backwards. So the top layers overfit, and the lower layers don't get tuned effectively. The lower layers essentially fire random noise, and the top layers overfit. For this reason, before 2006, no one knew how to train a deep architecture (besides Yann LeCun, with his convolutional architectures, but those were not as general purpose). The breakthrough in 2006 came when Hinton came up with the original DBN algorithm. Bengio et al (http://www.iro.umontreal.ca/~lisa/publications/index.php?page=publication&kind=single&ID=190 http://www.iro.umontreal.ca/~lisa/publications/index.php?pag...) followed up by teasing apart the important steps in training a deep architecture. They were: greedy layerwise training, i.e. construct one layer, then the next layer, then the next layer, etc. unsupervised pretraining When you train a single layer using an unsupervised criterion, the gradient signal is passed backwards through only one hidden layer (the layer you are constructing). And the output layer for an unsupervised criterion has as many units as the input. So the output layer passes back a strong gradient signal, and it doesn't have to travel very far. By doing this unsupervised pretraining in a layer-by-layer manner, the deep network receives a good initialization of its parameters. Then, when finetuning against the supervised criterion using backprop, you can find a better local minimum. The work of Erhan et al show that the effect of unsupervised pretraining is not only regularization, but also improved optimization. Take a look at there work for a great empirical study with large scale experiments and pretty graphs: http://jmlr.csail.mit.edu/proceedings/papers/v9/erhan10a/erhan10a.pdf http://jmlr.csail.mit.edu/proceedings/papers/v9/erhan10a/erh... p.s. feel free to send me an email to followup.