4 ms·
No, because it's wrong. Threshold neurons are still differentiable almost everywhere, no different than Relus which are ubiquitous. They may not be very good ac
by shmageggy 5y ago
No, because it's wrong. Threshold neurons are still differentiable almost everywhere, no different than Relus which are ubiquitous. They may not be very good activation functions but they don't prevent a network from learning.
- sdenton4 5y agoSorry, but I think I disagree... If I understand the PM neuron, it's outputs are boolean. This means there's no backprop signal, despite differentiability, since the outputs are constant (thus, gradient zero) in any neighborhood, which in turns zeros out any learning signal you would want to backprop through them. So you need a different learning strategy than backprop to use them. (see also: the 'dead neuron' problem/phenomenon with ReLU activations.)
- orbifold 5y agoOne can regularise the derivative (a delta distribution) in several ways (e.g. a triangle at zero) and that is good enough (even from a theoretical perspective) to find an approximate gradient. Experimentally it is then possible to train deep neural networks with such non-linearities.