If you haven't, you should take a few moments to read the original AlexNet paper (only 11 pages): https://papers.nips.cc/paper/4824-imagenet-classification-wit

If you haven't, you should take a few moments to read the original AlexNet paper (only 11 pages): https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf https://papers.nips.cc/paper/4824-imagenet-classification-wi... What you're saying is true, it should have worked in theory, but it just wasn't working for decades. The AlexNet team made several critical optimizations to get it work: (a) big network, (b) training on GPU, and (c) using a ReLU instead of tanh(x). In the end, it was the hardware that made it possible, but up until their paper it really wasn't for sure. A good analogy is the invention of the airplane. You can speculate all you want about the curvature of a bird's wing and lift, but until you actual build a wing that flies, it's all speculation.