After reading the article, my summary of the message is not "backpropagation is a leaky abstraction" but instead "if you don't understand how the derivatives ar

After reading the article, my summary of the message is not "backpropagation is a leaky abstraction" but instead "if you don't understand how the derivatives are being calculated, it will come back to bite you". The author talks about issues that arise with s-curve activation functions having saturated outputs and minimized gradients (part of the historical reason for moving away from these functions), the vanishing/exploding gradient problem in RNNs caused by repeated multiplication, the issue with clipping gradients as a solution, and the issue with zero-valued gradients in ReLUs. This would be equivalent to being a DBA without understanding indexes, or being a web developer without understanding the DOM. Yes, you can get by for a while, coasting on your tools. But when the stakes are high and you need to get it right, that ignorance will hamstring you. You don't want to be put in the situation of building a NN for someone and having no good idea about why it isn't working yet.