What innovations do you have in mind? LSTMs and convnets were both invented in the 90s. Stochastic gradient descent, momentum, etc. have been around for decades

What innovations do you have in mind? LSTMs and convnets were both invented in the 90s. Stochastic gradient descent, momentum, etc. have been around for decades. I feel like it's quite the opposite of what you are saying -- in the 2000s there was a lot of innovation, but it mostly turned out to be unnecessary. What has improved is the variety of architectures, but I don't think there has been anything fundamentally new after LSTMs. The only exception I can think of is generative adversarial networks, and even ideas behind adversarial training have been around in other domains for many years now.