3 ms·

Tanh belongs to a class of functions called "smooth steps" which I guess was being abbreviated as "step." Obviously the derivative of a step is zero everywhere

• l33tman 5y ago
  An interesting thing is that ReLu (which is absolutely the most common activation function) has a zero derivative over half the space its defined, and backprop still works (most of the time :). There are leaky ReLU to give some slope to the negative side, but it seems in most normal deep learning networks this is not required, there are always some neurons that aren't in the off regime to "catch the error".