3 ms·
https://stats.stackexchange.com/questions/237169/why-are-non-zero-centered-activation-functions-a-problem-in-backpropagation https://stats.stackexchange.com/que
by kawhi_ 9y ago
https://stats.stackexchange.com/questions/237169/why-are-non-zero-centered-activation-functions-a-problem-in-backpropagation https://stats.stackexchange.com/questions/237169/why-are-non...
https://www.quora.com/What-is-the-issue-with-back-propagation-in-Neural-Networks-when-the-Activation-function-only-outputs-Positive-values https://www.quora.com/What-is-the-issue-with-back-propagatio...
- p1esk 9y agoOh ok, I see. He was talking about only the gradients of weights all going to the same, single neuron. Then yes, the gradient of the loss in respect to the output of that neuron is a single value, which will be multiplied by x vector. So w_grad vector of weights going to that neuron will have a sign of dL/df, which again, is a single value. Hopefully this is clear.