3 ms·
Note that the example here is forward differentiation and most machine learning these days is backward propagation
by dnautics 6y ago
Note that the example here is forward differentiation and most machine learning these days is backward propagation
- oxinabox 6y agoNote that this applies backwards also. Of the 7 ADs demoed, only 1 was forward. The other 6 were reverse mode. All gave same result
- dnautics 6y agoOf course, sorry didn't mean to imply that AD was useful only in one direction. Haha I guess I was caught only reading 1/7 of the article!!
- rockinghigh 6y agoBackward propagation computes the gradient of the loss function with respect to the weights. A gradient is a vector or tensor of derivatives. You need to differentiate the loss function somehow to evaluate this vector.
- dnautics 6y agoThat's not strictly speaking true. You also calculate the gradient of the input. Otherwise, you wouldn't be able to backpropagate over more than one layer. It's just useful because you can do both easily, where with forward mode getting derivatives with respect to weights is considerably more expensive.