3 ms·
Of course it's a mixture of positive and negative values. Why would you think otherwise?
by p1esk 9y ago
Of course it's a mixture of positive and negative values. Why would you think otherwise?
- kawhi_ 9y agohttps://stats.stackexchange.com/questions/237169/why-are-non-zero-centered-activation-functions-a-problem-in-backpropagation https://stats.stackexchange.com/questions/237169/why-are-non... https://www.quora.com/What-is-the-issue-with-back-propagation-in-Neural-Networks-when-the-Activation-function-only-outputs-Positive-values https://www.quora.com/What-is-the-issue-with-back-propagatio...
- p1esk 9y agoOh ok, I see. He was talking about only the gradients of weights all going to the same, single neuron. Then yes, the gradient of the loss in respect to the output of that neuron is a single value, which will be multiplied by x vector. So w_grad vector of weights going to that neuron will have a sign of dL/df, which again, is a single value. Hopefully this is clear.