6 ms·
Differentiable Programming – A Simple Introduction
- fennecs 4y agoDoes someone have an example where the ability to “differentiate” a program gets you something interesting? I understand perfectly what it means for a neural network, but how about more abstract things. Im not even sure as currently presented, the implementation actually means something. What is the derivative of a function like List, or Sort or GroupBy etc? These articles all assume that somehow it just looks like derivative from calculus somehow. Approximating everything as some non smooth real function doesn’t seem entirely morally correct. A program is more discrete or synthetic. I think it should be a bit more algebraic flavoured, like differentials over a ring.
- PartiallyTyped 4y agoThe nice thing about differentiable programming is that we can use all sorts of different optimizers compared to gradient descent that can offer quadratic convergence instead of linear!
- SleekEagle 4y agoYes exactly! This is huge. Hessian optimization is really easy with JAX, haven't tried it in Julia though
- PartiallyTyped 4y agoAnd very fast given that you compile the procedure! I am considering writing an article on this and posting it here because I have seen enormous improvements over non jitted code, and that excluded jax.vmap.
- SleekEagle 4y agoThere's a comparison of JAX with PyTorch for Hessian calculation here! https://www.assemblyai.com/blog/why-you-should-or-shouldnt-be-using-jax-in-2022/#hessians https://www.assemblyai.com/blog/why-you-should-or-shouldnt-b... Would definitely be interested in an article like that if you decide to write it
- ChrisRackauckas 4y agoHere's Hessian-Free Newton-Krylov on neural ODEs with Julia: https://diffeqflux.sciml.ai/dev/examples/second_order_adjoints/ https://diffeqflux.sciml.ai/dev/examples/second_order_adjoin... . It's just standard tutorial stuff at this point.
- applgo443 4y agoWhy can't we use this quadratic convergence in deep learning?
- PartiallyTyped 4y agoWell, quadratic convergence usually requires the Hessian, or an approximation of it, and that's difficult to get in deep learning due to memory constrains, and difficulty computing second order derivatives. Computing the derivatives is not very difficult with e.g. Jax, but ... you get back to the memory issue. The Hessian is a square matrix, so in Deep Learning, if we have a million of parameters, then the Hessian is a 1 trillion square matrix...
- tome 4y agoNot only does it have 1 trillion elements, you also have to invert it!
- SleekEagle 4y agohttps://c.tenor.com/enoxmmTG1wEAAAAC/heart-attack-in-pain.gif https://c.tenor.com/enoxmmTG1wEAAAAC/heart-attack-in-pain.gi...
- PartiallyTyped 4y agoIndeed! BFGS (and derivatives) approximate the inverse but they have other issues that make them prohibitively expensive.
- ssivark 4y agoTo add, one could think of schemes like "momentum" and cousins as attempts to estimate something in the spirit of the inverse Hessian using various hacks/heuristics.
- yauneyz 4y agoMy professor has talked about this. He thinks that the real gem of the deep learning revolution is the ability to take the derivative of arbitrary code and use that to optimize. Deep learning is just one application of that, but there are tons more.
- SleekEagle 4y agoThat's part of why Julia is so exciting! Building it specifically to be a differentiable programming language opens so many doors ...
- mountainriver 4y agoJulia wasn’t really built specifically to be differentiable, it was just built in a way that you have access to the IR, which is what zygote does. Enzyme AD is the most exciting to me because any LLVM language can be differentiable
- SleekEagle 4y agoAh I see, thank you for clarifying. And thank you for bringing Enzyme to my attention - I've never seen it before!
- celrod 4y agoEnzyme.jl works quite well (but the possibility of using it across languages is appealing).
- melony 4y agoI am just happy that the previously siloed fields of operations research and various control theory sub-disciplines are now incentivized to pool their research together thanks to the funding in ML. Also many expensive and proprietary optimization software in industry are finally getting some competition.
- 4y ago
- infogulch 4y agoThe most interesting thing I've seen on AD is "The simple essence of automatic differentiation" (2018) [1]. See past discussion [2], and talk [3]. I think the main idea is that by compiling to categories and pairing up a function with its derivative, the pair becomes trivially composable in forward mode, and the whole structure is easily converted to reverse mode afterwards. [1]: https://dl.acm.org/doi/10.1145/3236765 https://dl.acm.org/doi/10.1145/3236765 [2]: https://news.ycombinator.com/item?id=18306860 https://news.ycombinator.com/item?id=18306860 [3]: Talk at Microsoft Research: https://www.youtube.com/watch?v=ne99laPUxN4 https://www.youtube.com/watch?v=ne99laPUxN4 Other presentations listed here: https://github.com/conal/essence-of-ad https://github.com/conal/essence-of-ad
- tome 4y ago> the whole structure is easily converted to reverse mode afterwards. Unfortunately it's not. Elliot never actually demonstrates in the paper how to implement such an algorithm, and it's very hard to write compiler transformations in "categorical form". (Disclosure: I'm the other of another paper on AD.)
- orbifold 4y agoI think JAX effectively demonstrates that this is indeed possible. The approach they use is to first linearise the JAXPR and then transpose it, pretty much in the same fashion as the Elliot paper did.
- tome 4y agoA JAXPR is a normal functional-style expression tree, with free variables, like let b = f a in g (a, b) Elliot's "compiling to categories" requires you to translate this to g ∘ (id × f) ∘ dup It's pretty baffling to work with terms like the latter in practice! The main ideas that JAX is based on were already published in "Lambda the ultimate backpropagator" in 2008. Elliot's work is a nice way of conceptualising the AD transformations and understanding how they all relate to each other, but it's not particularly practical.
- 4y ago
- choeger 4y agoNice article, but the intro is a little lengthy. I have one remark, though: If your language allows for automatic differentiation already, why do you bother with a neural network in the first place? I think you should have a good reason why you choose a neural network for your approximation of the inverse function and why it has exactly that amount of layers. For instance, why shouldn't a simple polynomial suffice? Could it be that your neural network ends up as an approximation of the Taylor expansion of your inverse function?
- SleekEagle 4y agoI think for more complicated examples like RL control systems a neural network is the natural choice. If you can incorporate physics into your world model then you'd need differentiable programming + NNs, right? Or am I misunderstanding the question. If you're talking about the specific cannon problem, you don't need to do any learning at all you can just solve the kinematics, so in some sense you could ask why you're using any approximation function,
- simulate-me 4y agoRequiring an exact justification for a specific NN architecture is not productive. Simply put, such justification almost never exists, yet NN clearly out-perform simple polynomials in a wide variety of tasks. Even if you know that a NN performs better than a simple polynomial on a task, you're very unlikely to get an exact explanation on why a specific NN was chosen vs. the infinite number of other possible NNs. If you require such an explanation, then you're going to miss out on a lot of performance.
- potatoman22 4y agoMost of the time we hear about neural networks because the linear models didn't work.
- adgjlsfhk1 4y agoin low dimensional spaces, there are a lot of good techniques, but once you go above 10 or so, NN is generally the only option. Pretty much everything else has exponential degradation with more dimensions.
- noobermin 4y agoThe article is okay but it would have helped to have labelled the axes of the graphs.
- fghorow 4y agoAt first glance, this approach appears to re-invent an applied mathematics approach to optimal control. There, one writes a generalized Hamiltonian, from which forward and backward-in-time paths can be iterated. The Pontryagin maximum (or minumum, if you define your objective function with a minus sign) principle is the essence to that approach to optimal control.
- SleekEagle 4y agoI've never heard of Pontryagin's maximum principle, but thanks for bringing it to my attention. I think my knowledge of dynamical systems and control theory isn't quite up-to-snuff at the moment to fully understand it, but it's bookmarked for another day! thanks again