It is different though. The derivative of the L2 loss function w.r.t. the linear regression parameters is a "flat" function that is easy to derive by hand. With

It is different though. The derivative of the L2 loss function w.r.t. the linear regression parameters is a "flat" function that is easy to derive by hand. With neutral networks you have deeply nested vector-valued functions. If you write down the chain rule, it suggests that you should compute the Jacobians of all the nested vector valued functions and then multiply them together. This would be computationally expensive. The key idea of backpropagation is that at each layer, you only ever need the derivatives of the loss function w.r.t. the parameters of the layer, and the Jacobian-vector product with the derivatives of the loss function w.r.t. the layer outputs. You never need to compute the jacobians explicitly and you never need to do those high dimensional matrix-matrix multiplications. These are not complicated ideas but they involve a combination of software design, calculus, and linear algebra that would probably not be obvious to the average CS undergrad.