In my opinion fundamental NN theory hasn’t really advanced significantly since the 1990s. I guess it was phrased ambiguously in my comment But there has been a

In my opinion fundamental NN theory hasn’t really advanced significantly since the 1990s. I guess it was phrased ambiguously in my comment But there has been a great deal of new techniques that make NN work better in practice like dropout, ReLU for vanishing gradients, CNNs and GANs for specific problem types, transfer learning, etc. The recent work with Neural ODEs show that the field is advancing in terms of ideas. This is a good trajectory IMO. Many engineering fields work this way — find out new ways of doing things that work and then take a step back to see if there’s anything fundamental that links everything together. Practice precedes theory.