4 ms·

> They can't learn Is there a good (i.e. theoretical) reason for this?

• Arech 5y ago
  Non-differentiability, obviously
• shmageggy 5y ago
  No, because it's wrong. Threshold neurons are still differentiable almost everywhere, no different than Relus which are ubiquitous. They may not be very good activation functions but they don't prevent a network from learning.
  • sdenton4 5y ago
    Sorry, but I think I disagree... If I understand the PM neuron, it's outputs are boolean. This means there's no backprop signal, despite differentiability, since the outputs are constant (thus, gradient zero) in any neighborhood, which in turns zeros out any learning signal you would want to backprop through them. So you need a different learning strategy than backprop to use them. (see also: the 'dead neuron' problem/phenomenon with ReLU activations.)
    • orbifold 5y ago
      One can regularise the derivative (a delta distribution) in several ways (e.g. a triangle at zero) and that is good enough (even from a theoretical perspective) to find an approximate gradient. Experimentally it is then possible to train deep neural networks with such non-linearities.