3 ms·
Why is the gradient either +ve or -ve when we use sigmoid activation function?
whenever i ask this question people tell me something like
""Gradient of W = X * gradient of Function
Now, from the above equation, since all X values are positive, therefore gradient of all W's could either be all positive or negative, depending on the gradient of function.""
i get it but i think they are missing a point.
consider the multi-class setting - if there are 10 classes, and we use a softmax output. The NN makes a prediction C1, whereas the true class is C2. Now our gradient of the ouput layer(from the loss function) will be a vector which will have one positive term, and another negative term (for C1 and C2). When this vector is multiplied with positive gradients during backprop. Since error gradient has both positive and negative numbers, the gradients of W should also be a mix of positive and negative.
NOT JUST THAT...
also the gradient over the hidden vectors dhidden when backpropagated will be multiplied by weights which can have positive and negative signs.
I am i right?
- p1esk 9y agoOf course it's a mixture of positive and negative values. Why would you think otherwise?
- kawhi_ 9y agohttps://stats.stackexchange.com/questions/237169/why-are-non-zero-centered-activation-functions-a-problem-in-backpropagation https://stats.stackexchange.com/questions/237169/why-are-non... https://www.quora.com/What-is-the-issue-with-back-propagation-in-Neural-Networks-when-the-Activation-function-only-outputs-Positive-values https://www.quora.com/What-is-the-issue-with-back-propagatio...
- p1esk 9y agoOh ok, I see. He was talking about only the gradients of weights all going to the same, single neuron. Then yes, the gradient of the loss in respect to the output of that neuron is a single value, which will be multiplied by x vector. So w_grad vector of weights going to that neuron will have a sign of dL/df, which again, is a single value. Hopefully this is clear.