4 ms·
Isn't the input of the layer fundamentally a part of the gradient computation? So even in this case (inspecting LLVM code) the computation still needs to look a
by cl3misch 6y ago
Isn't the input of the layer fundamentally a part of the gradient computation? So even in this case (inspecting LLVM code) the computation still needs to look at the input.
- wsmoses 6y agoYou don't always need the input to compute the gradient. For example the gradient of a sum function doesn't require the original input, it just sets all of the derivative(input)'s to 1.
- shoyer 6y agoTo be more precise, in backwards mode auto-diff, inputs only need to be saved if they are used in a non-linear way.
- PartiallyTyped 6y agoEven so, you maybe able to perform optimizations that were not possible under normal circumstances, e.g. you have an exponent in the output of a layer followed by a log in the next. Think SoftMax and logloss.