To expand on what gcp is saying, that's not true. Rectified linear units do better than sigmoids but are not differentiable (but are sub-differentiable), and Re

To expand on what gcp is saying, that's not true. Rectified linear units do better than sigmoids but are not differentiable (but are sub-differentiable), and ReLUs have the problem where small changes don't do anything (in the negative regime).