5 ms·
Differentiable Programming: A Semantics Perspective
- vutekst 9y agoArticle is a bit broken for me because while the page is https, the mathjax file is loaded from a plain http. This is apparently forbidden by current Chrome and Firefox, so the figures and LaTeX don't work. edit: ah-hah, HTTPS Everywhere must have fiddled with it.
- seanmcdirmid 9y agoA decade ago, symbolic differentiation could also be used to compute normal vectors for mathematically defined surfaces (like iso surfaces). Conal Elliott (the guy who co-invented FRP) wrote a great paper on this with fairly well defined semantics: http://conal.net/Vertigo/ http://conal.net/Vertigo/ And http://conal.net/papers/beautiful-differentiation/ http://conal.net/papers/beautiful-differentiation/
- alanbernstein 9y agoThat's neat. I'm not sure I see the connection, other than the use of differentiation, broadly speaking.
- davidwihl 9y agoAutomatic differentiation is a little different than symbolic differentiation. Rather than use a new high level description of the computational graph, tools like autograd [1] can differentiate somewhat arbitrary Python and numpy instructions. [1] https://github.com/HIPS/autograd https://github.com/HIPS/autograd
- mehrdadn 9y ago> can differentiate somewhat arbitrary Python and numpy instructions. I hear this a lot but it doesn't make sense to me. How do you differentiate an conditional or a loop? It doesn't make sense mathematically, so I'm not sure what sense it makes programmatically.
- improbable22 9y agoYou regard a conditional is a piecewise smooth function, like f(x) = {0 if x<0, x^2 if x>0}. Then the derivative is that of the smooth piece you evaluated. Or to say that another way, since the derivative by definition changes x infinitesimally, it never changes on which side of the conditional x is located.
- mehrdadn 9y agoWhat if it was the following, with x = 0? y = x if x > 0 else -2 * x if x < 0 else 0
- OisinMoran 9y agoFor functions that are not differentiable you can use the subdifferential [0] (or subgradient) which gives a set rather than a single value. In your example this would be the interval [-2,1]. In practice, from what I've read it seems to be fine to just take any value between these and most commonly they are just averaged. It helps that these regions are rarely encountered. For more detail, check out Chapter 6 [1], Section 6.3 "Hidden Units" in The Deep Learning book. [0] https://en.wikipedia.org/wiki/Subderivative https://en.wikipedia.org/wiki/Subderivative [1] http://www.deeplearningbook.org/contents/mlp.html http://www.deeplearningbook.org/contents/mlp.html
- mehrdadn 9y agoNope. I thought about subderivative too... but then I realized this isn't even true. Constructing a counterexample is trivial: y = x if x > 0 else x if x < 0 else 0 Zero is in no way a subderivative at x = 0.
- OisinMoran 9y agoYeah you're right, the subdifferential of f(x)=|x| at the origin is the interval [−1, 1]. In the fourth paragraph of link [1] in the above post they explain quite well why we can get away with this type of trickery even if it seems wrong.
- seanmcdirmid 9y agoIf one defines a subset DSL that can be auto differentiated via a symbolic method, then is it really that different? Embedded DSLs are pretty common in the Haskell world anyways.
- noelwelsh 9y agoSymbolic differentiation doesn't work for loops and conditionals. The expression of the derivative is also exponential in the size of the source expression.
- seanmcdirmid 9y agoOk, I got it. Just as a clarification, Conal is talking about auto differentiation also. This was my mistake in presenting his work.
- catnaroek 9y agoI don't understand in what sense programs can be called “differentiable”. Is the space of programs modulo observational equivalence a manifold to begin with? (I don't think it's Hausdorff or even T1, but I could be wrong.) The examples given in the article are merely derivatives of ordinary mathematical functions defined by ordinary mathematical expressions - in particular, there are no sequencing, no conditionals and no loops. So why call them “differentiable programs” when you are actually dealing with ordinary differentiable functions from good old 19th century analysis? We need urgent improvements in the intellectual honesty department.