3 ms·

What really made the difference was non-linear activations. Without a non-linearity depth doesn’t buy you anything.

by enchiridion 5y ago

visarga 5y ago
It took us long enough to settle on ReLu. Step function activations were brutal.
- canjobear 5y ago
  I seem to recall that before ReLU it was usually tanh.
  - whatshisface 5y ago
    Tanh belongs to a class of functions called "smooth steps" which I guess was being abbreviated as "step." Obviously the derivative of a step is zero everywhere it's defined so backprop wouldn't work.
    - l33tman 5y ago
      An interesting thing is that ReLu (which is absolutely the most common activation function) has a zero derivative over half the space its defined, and backprop still works (most of the time :). There are leaky ReLU to give some slope to the negative side, but it seems in most normal deep learning networks this is not required, there are always some neurons that aren't in the off regime to "catch the error".
  - _0ffh 5y ago
    One of a whole class of functions called sigmoidal, usally the sigmoid or the tanh function. These activation functions dominated the field for a very long time.
l33tman 5y ago
Interesting counterpoint: a NN is not just activation. It's also learning, and multilayer stacks learn differently depending on depth even if they are entirely linear. This seems to be an artefact due to the optimizer in use (SGD for example). The jury seems to be out still on why this is.. the issue has been known since long, and I recently saw a short paper Yann LeCun posted about exactly this as well, they stacked a bunch of completely linear layers on top of a "normal" DL stack and got differences in the final system, even though you can collapse all the linear layers at any time to a single linear mixing layer.
greenbit 5y ago
Truth! Without nonlinearity, each layer is essentially just a matrix multiplication. You could just as well use a single equivalent layer that represents the product of the layers' matrices. Nonlinearity lets a single node partition the input space into two (fuzzy) equivalence classes. That's powerful stuff.

What really made the difference was non-linear activations. Without a non-linearity depth doesn’t buy you anything.

visarga 5y ago

It took us long enough to settle on ReLu. Step function activations were brutal.

canjobear 5y ago
I seem to recall that before ReLU it was usually tanh.
- whatshisface 5y ago
  Tanh belongs to a class of functions called "smooth steps" which I guess was being abbreviated as "step." Obviously the derivative of a step is zero everywhere it's defined so backprop wouldn't work.
  - l33tman 5y ago
    An interesting thing is that ReLu (which is absolutely the most common activation function) has a zero derivative over half the space its defined, and backprop still works (most of the time :). There are leaky ReLU to give some slope to the negative side, but it seems in most normal deep learning networks this is not required, there are always some neurons that aren't in the off regime to "catch the error".
- _0ffh 5y ago
  One of a whole class of functions called sigmoidal, usally the sigmoid or the tanh function. These activation functions dominated the field for a very long time.

l33tman 5y ago

Interesting counterpoint: a NN is not just activation. It's also learning, and multilayer stacks learn differently depending on depth even if they are entirely linear. This seems to be an artefact due to the optimizer in use (SGD for example). The jury seems to be out still on why this is.. the issue has been known since long, and I recently saw a short paper Yann LeCun posted about exactly this as well, they stacked a bunch of completely linear layers on top of a "normal" DL stack and got differences in the final system, even though you can collapse all the linear layers at any time to a single linear mixing layer.

greenbit 5y ago

Truth! Without nonlinearity, each layer is essentially just a matrix multiplication. You could just as well use a single equivalent layer that represents the product of the layers' matrices. Nonlinearity lets a single node partition the input space into two (fuzzy) equivalence classes. That's powerful stuff.