So, what should we do instead? The author gives some recommendations: "Research to make deep learning easier to use has a huge impact, making it faster and sim

So, what should we do instead? The author gives some recommendations: "Research to make deep learning easier to use has a huge impact, making it faster and simpler to train better networks. Examples of exciting discoveries that have now become standard practice are: * Dropout allows training on smaller datasets without over-fitting. * Batch normalization allows for faster training. * Rectified linear units help avoid gradient explosions. Newer research to improve ease of use includes: * The learning rate finder makes the training process more robust. * Super convergence speeds up training, requiring fewer computational resources. * “Custom heads” for existing architectures (e.g. modifying ResNet, which was initially designed for classification, so that it can be used to find bounding boxes or perform style transfer) allow for easier architecture reuse across a range of problems. None of the above discoveries involve bare-metal power; instead, all of them were creative ideas of ways to do things differently." These seem quite fundamental (except for the last one which sounds like a favorite domain adaptation strategy from the author). I'm in industry myself, but it's quite hard to come up with fundamental strategies. In particular I'm trying to merge nonparametric Bayesian models with deep networks. By enriching the latent variables in autoencoders to richer priors we might see improvements. Subsequently, we simultaneously need better control variates to do inference in more complex models. See my blog post on the overlooked topic of control variates: https://www.annevanrossum.com/blog/2018/05/26/random-gradients/ https://www.annevanrossum.com/blog/2018/05/26/random-gradien.... If we really want to be creative we need people from academia on board.