>For example, we now know that deep neural nets can be trained with approximate gradients obtained by shifting bits to get the sign and order of magnitude of th

>For example, we now know that deep neural nets can be trained with approximate gradients obtained by shifting bits to get the sign and order of magnitude of the gradient roughly right.[1] In some cases it's even possible to restrict learning to use binary weights.[2] More recently, we have learned that it's possible to use "helper" linear models during training to predict what the gradients will be for each layer, in-between true-gradient updates, allowing layers to update their parameters locally during backpropagation.[3] Finally, don't forget that in the late 2000's, AI researchers were doing a lot of interesting work with unsupervised layer-wise training (e.g., DBNs composed of RBMs, stacked autoencoders).[4] All of those use fundamentally use back propagation though, they just approximate the gradients. Pretraining with autoencoders even used back propagation to train the autoencoders. And researchers have entirely moved away from that strategy because it just doesn't work as well as pure supervised learning. I don't think backpropagation will ever go away because it is so simple and powerful. Instead it will be tweaked and approximated like in your examples.