Except that you are talking about the derivative with respect to each weight. In a deep neural network, you'd need several sample points for each weight in ord

Except that you are talking about the derivative with respect to each weight. In a deep neural network, you'd need several sample points for each weight in order to compute dC/dw. That is a LOT of work for a single step. The only way to do this in a timely fashion is via backprop, which requires a derivative.