From a mostly outside point of view, I've wondered a lot about this. It seems a lot of ML/DL research that goes on nowadays amounts to educated guessing and ti

From a mostly outside point of view, I've wondered a lot about this. It seems a lot of ML/DL research that goes on nowadays amounts to educated guessing and tinkering i.e. "Based on our notion of what seemed right, we tried this other configuration of layers and/or activation functions and look; it worked". I imagine at some point that the kind of research discussed in the linked post will begin to pan out more and more and will eventually lead to another period of rapid progress like that which followed Hinton's work on DL.