That is not true, one can easily regularise the derivative, which is delta distribution, in some appropriate way, the easiest one being a 'triangle' centered at

That is not true, one can easily regularise the derivative, which is delta distribution, in some appropriate way, the easiest one being a 'triangle' centered at zero. That way one can actually easily train networks of McCulloch-Pitts neurons.