5 ms·
"The second factor, and probably the more damning one, is that most ML codes don't actually use that much dynamism." I would argue that this in true precisely b
by spacetracks 5y ago
"The second factor, and probably the more damning one, is that most ML codes don't actually use that much dynamism." I would argue that this in true precisely because it is not available in an AD system. When I tell friends and coworkers about what zygote can do they light up and start describing different use cases they have that could benefit from AD. Diff eq solving is a big one.
- KKKKkkkk1 5y agoThis is because continuous optimization is useless when crossing a discontinuity, which is what control flow creates. Even in a trivial situation like ReLU, where the control flow is mimicking a continuous transition, you have the "dead ReLU" problem, where you have to start training on the correct side of the discontinuity and make sure to never cross.
- ogogmad 5y agoI don't know whether this belongs here, but... Formally, there is a generalisation of differentiation which can handle functions like ReLU (i.e. locally Lipschitz non-differentiable functions) by allowing a derivative to be set-valued. It's called the Clarke gradient. The Clarke gradient of ReLU at 0 is the closed interval [0,1]. Note that the Clarke gradient doesn't satisfy the chain rule (except in a weakened form) which might seriously mess up some assumptions about autodiff. Is this generalised derivative useful in autodiff? I imagine that this is a largely theoretical tool that's useful in analysing algorithms but useless for actually computing things.
- medo-bear 5y agoi havent heard of clarke gradient before. convex analysis has something called subgradient [0], is it different? [0] https://en.m.wikipedia.org/wiki/Subderivative https://en.m.wikipedia.org/wiki/Subderivative
- ogogmad 5y agoThe subgradient in convex analysis is a special case of the Clarke gradient. The subgradient is precisely the Clarke gradient for convex functions. Convex functions are always locally Lipschitz except in weird cases. [edit] Question: Are there numerical applications in which the subgradient is actually computed, or is it a purely analytical tool?
- agnosticmantis 5y ago(Stochastic) subgradient methods are used in practice to optimize non-differentiable convex functions. They have a slower convergence rate than (stochastic) gradient descent though. See for example : https://www.stat.cmu.edu/~ryantibs/convexopt-F15/lectures/07-sg-method.pdf https://www.stat.cmu.edu/~ryantibs/convexopt-F15/lectures/07...
- fault1 5y agoyes, see: https://juliadiff.org/ChainRulesCore.jl/dev/maths/nondiff_points.html https://juliadiff.org/ChainRulesCore.jl/dev/maths/nondiff_po... and related neat usages in set based optimization methods in MathOptInterface (part of JuMP.jl): https://matbesancon.xyz/post/2020-12-24-chains_sets2/ https://matbesancon.xyz/post/2020-12-24-chains_sets2/
- ogogmad 5y agoTo me, it looks like you're computing a single element from the set, instead of the whole set. Which makes sense, but it's an important caveat.
- eigenspace 5y agoThis is a problem in theory, but not really in practice for a well designed AD system. We wrote a bit about this here: https://juliadiff.org/ChainRulesCore.jl/dev/maths/nondiff_points.html https://juliadiff.org/ChainRulesCore.jl/dev/maths/nondiff_po... The gist of it is that we endeavour to provide ‘useful’ values for the gradient at non-differentiable points, even if the traditional derivative is not defined or infinite. You could imagine it as us smoothing out edges or discontinuities, ideally in a way that that makes things like gradient descent well behaved.