For neural networks to work, the activation function has to be nonlinear and monotonic. In order to be able to apply the chain rule for training the weights by

For neural networks to work, the activation function has to be nonlinear and monotonic. In order to be able to apply the chain rule for training the weights by backpropagation, the function also has to be differentiable. The logistic function is one obvious choice, but other functions such as tanh, arctan or relu are popular as well.