3 ms·
I'm not familiar with the details of backprop in neural networks, but AIUI it's an application of automatic/algorithmic differentiation, which comes in two mode
by omnicognate 5d ago
I'm not familiar with the details of backprop in neural networks, but AIUI it's an application of automatic/algorithmic differentiation, which comes in two modes: forward and reverse.
Reverse mode is harder to implement as you need to retain state through the calculation, but it scales differently. Forward mode is O(number of inputs) while reverse is O(number of outputs). Seems obvious that reverse mode is what you want for training a neural network, where you have huge numbers of inputs and usually one output, the loss you're training on.
(And indeed that appears to be what the article is saying, in different language.)
- _0ffh 5d agoTrue, though the timeline was more the other way around. Error backprop was used as a method for training before using AD to automate the implementation work became the norm. Previously you had to write the backward pass by hand, now we use AD to derive it from the forward pass.
- omnicognate 5d agoInteresting, didn't know that history. I've used algo diff but in unrelated fields. I think it's a technique that has been independently discovered several times and has terribly inconsistent terminology across domains as a result.