Not necessarily. It turns out to really matter how the weights are initialized (randomly, but from which distribution?) for ordinary feed-forward neural nets t

Not necessarily. It turns out to really matter how the weights are initialized (randomly, but from which distribution?) for ordinary feed-forward neural nets that only uses standard neurons -- different activation functions have different optimal initialization distributions. https://machinelearningmastery.com/weight-initialization-for-deep-learning-neural-networks/ https://machinelearningmastery.com/weight-initialization-for... A big breakthrough was Kaiming He's work on initialization for ReLU: https://arxiv.org/abs/1502.01852 https://arxiv.org/abs/1502.01852 (Kaiming is the personal name, He is the family name. He Kaiming is the native Chinese order. Lots of citations use "Kaiming" by mistake.)