Part of the issue with neural nets is that historically they were next to impossible to train. ADAM, BatchNorm/LayerNorm, initialization schemes, and GPUs for

Part of the issue with neural nets is that historically they were next to impossible to train. ADAM, BatchNorm/LayerNorm, initialization schemes, and GPUs for pure speed really helped to change all of that.