You’re confusing “everywhere differentiable” with “differentiable at the points we care about.” g(x) = abs(x) is differentiable almost everywhere (not in the Ca

You’re confusing “everywhere differentiable” with “differentiable at the points we care about.” g(x) = abs(x) is differentiable almost everywhere (not in the Cantor’s Staircase sense, but in a practical sense), and that’s enough for gradient descent to work in practice. And fitting models by gradient-descent-like methods is the single unifying feature of deep learning. I don’t understand the implementation thing you’re worrying about. Assuming it takes the same inputs to the same outputs, how you implement a function has no bearing on its limit properties, which is what differentiability is about.