Is that actually true? From my cursory understanding, it seems that there has been a slew of theoretical breakthroughs (including ways to architecte and train n

Is that actually true? From my cursory understanding, it seems that there has been a slew of theoretical breakthroughs (including ways to architecte and train networks aside from just blindly throwing more computation resources) that significantly pushed the field forward. Off the top of my head, I'm thinking of Hinton's 2006 DNN which pre-trained each layer one at a time as a restricted Botlzmann machine, computationally simpler activation functions like ReLu, the development of residual networks, the development of GANs, etc. Of course GPU-based training has helped a lot, but my impression is that these theoretical breakthroughs were also important.