The inner activation function (AF) of neurons is inherently nonlinear; it has to be in order to solve any problem that is not linearly decomposable (which is ba

The inner activation function (AF) of neurons is inherently nonlinear; it has to be in order to solve any problem that is not linearly decomposable (which is basically all of the interesting problems). Often the AF nonlinearity shows up as a thresholding operation following a linear weighted sum, but that's not the only mechanism. And yet neurons are not "pure" binary thresholders the way logic gates are because you can't take the derivative of a binary function, and you can only do backpropagation on differentiable functions. The compromise neurons make is a "smoothed threshold" or sigmoidal curve which is differentiable but still very nonlinear.