An interesting thing is that ReLu (which is absolutely the most common activation function) has a zero derivative over half the space its defined, and backprop

An interesting thing is that ReLu (which is absolutely the most common activation function) has a zero derivative over half the space its defined, and backprop still works (most of the time :). There are leaky ReLU to give some slope to the negative side, but it seems in most normal deep learning networks this is not required, there are always some neurons that aren't in the off regime to "catch the error".