6 ms·
The things that have most improved training neural networks since you left were: 1. Smart (Xavier/He) initialization 2. ReLU activations 3. Batch normalization
by chillee 9y ago
The things that have most improved training neural networks since you left were: 1. Smart (Xavier/He) initialization 2. ReLU activations 3. Batch normalization 4. Residual connections
The GPUs and dataset size were definitely very important though.
- The_suffocated 9y agoThanks for the info. I may want to keep up with the recent advances some day.