4 ms·

I've derived backpropagation by hand many times and diagrams like these often just confuse me more. For me it depends on whether I'm in a passive or active sta

I've derived backpropagation by hand many times and diagrams like these often just confuse me more. For me it depends on whether I'm in a passive or active state of learning. If I'm sitting down on a Sunday afternoon reading the news, backpropagation is going to make zero sense to me. But, if I'm actively working on a problem, it's much more useful to realize that this is no different than using gradient descent for linear regression or even minimizing a quadratic. At that point, it becomes just a mechanical calculation (the fact that the resulting gradient looks more intimidating is irrelevant). And for me, realizing that it's no different than taking the derivative of a quadratic actually makes it more digestible than these fancy animated tutorials.

p1esk 8y ago
no different than using gradient descent for linear regression If you understand that already, then you are really not the target audience of blog posts like that.
blt 8y ago
It is different though. The derivative of the L2 loss function w.r.t. the linear regression parameters is a "flat" function that is easy to derive by hand. With neutral networks you have deeply nested vector-valued functions. If you write down the chain rule, it suggests that you should compute the Jacobians of all the nested vector valued functions and then multiply them together. This would be computationally expensive. The key idea of backpropagation is that at each layer, you only ever need the derivatives of the loss function w.r.t. the parameters of the layer, and the Jacobian-vector product with the derivatives of the loss function w.r.t. the layer outputs. You never need to compute the jacobians explicitly and you never need to do those high dimensional matrix-matrix multiplications. These are not complicated ideas but they involve a combination of software design, calculus, and linear algebra that would probably not be obvious to the average CS undergrad.
- p1esk 8y ago
  Here's a very good explanation of the whole thing, that should be accessible to any average high school student who bothered to take a calculus course: http://neuralnetworksanddeeplearning.com/ http://neuralnetworksanddeeplearning.com/

p1esk 8y ago

no different than using gradient descent for linear regression If you understand that already, then you are really not the target audience of blog posts like that.

blt 8y ago

It is different though. The derivative of the L2 loss function w.r.t. the linear regression parameters is a "flat" function that is easy to derive by hand. With neutral networks you have deeply nested vector-valued functions. If you write down the chain rule, it suggests that you should compute the Jacobians of all the nested vector valued functions and then multiply them together. This would be computationally expensive. The key idea of backpropagation is that at each layer, you only ever need the derivatives of the loss function w.r.t. the parameters of the layer, and the Jacobian-vector product with the derivatives of the loss function w.r.t. the layer outputs. You never need to compute the jacobians explicitly and you never need to do those high dimensional matrix-matrix multiplications. These are not complicated ideas but they involve a combination of software design, calculus, and linear algebra that would probably not be obvious to the average CS undergrad.

p1esk 8y ago
Here's a very good explanation of the whole thing, that should be accessible to any average high school student who bothered to take a calculus course: http://neuralnetworksanddeeplearning.com/ http://neuralnetworksanddeeplearning.com/