3 ms·
> It’s like chaining up perceptrons hoping to get more expressive power for free. Isn't that literally the cause of the success of deep learning? It's not quit
by falcor84 2y ago
> It’s like chaining up perceptrons hoping to get more expressive power for free.
Isn't that literally the cause of the success of deep learning? It's not quite "free", but as I understand it, the big breakthrough of AlexNet (and much of what came after) was that running a larger CNN on a larger dataset allowed the model to be so much more effective without any big changes in architecture.
- david2ndaccount 2y agoWithout a non-linear activation function, chaining perceptrons together is equivalent to one large perceptron.
- xanderlewis 2y agoYep. falcor84: you’re thinking of the so-called ‘multilayer perceptron’ which is basically an archaic name for a (densely connected?) neural network. I was referring to traditional perceptrons.
- falcor84 2y agoWhile ReLU is relatively new, AI researchers have been aware of the need for nonlinear activation functions and building multilayer perceptrons with them since the late 1960s, so I had assumed that's what you meant.
- xanderlewis 2y agoIt was a deliberately historical example.