In the early days (pre 2015) it was extremely difficult to train models with more than two or three layers. Batch Norm and Residual connections opened the door

In the early days (pre 2015) it was extremely difficult to train models with more than two or three layers. Batch Norm and Residual connections opened the door to actual depth. (On edit: ReLu activations were also vital for dealing with vanishing gradients. And there were some real advances in initialization and opportunities along the way, as well.)