That's arguable; most successful applications of deep learning aren't particularly efficient on the training side, which is why they come out of research groups

That's arguable; most successful applications of deep learning aren't particularly efficient on the training side, which is why they come out of research groups that have Google-sized compute clusters available, typically with big arrays of recent GPUs. I suppose they're efficient in the sense of not requiring more computation than exists on earth. There have been a bunch of algorithmic advances as well, but some advances have simply been those increases in compute power: there are techniques that didn't really work in 1985, where the technique or a close variant now works, mainly because we didn't the big arrays of GPUs in 1985 that we do now. As these get scaled up, the algorithmic advances are being found to not even be necessary in some cases. For a period it was thought that autoencoder pretraining was the big algorithmic breakthrough that made it feasible to train many-layered neural networks, but a number of recent applications no longer bother with the autoencoder pretraining.