3 ms·

Just gave a brief run through the paper, and I'm curious: is reverse-mode differentiation in this context similar in concept to adjoint-type methods used in com

by thearn4 11y ago

Just gave a brief run through the paper, and I'm curious: is reverse-mode differentiation in this context similar in concept to adjoint-type methods used in computational fluid dynamics (and increasingly, in design optimization in other fields)? Backwards-propagation of a gradient calculation to allow for extremely high-dimensional parameter spaces is a trick that has been around for awhile in some circles, but also seems to have been missed in a lot of other disciplines. I'm seeing a lot of publications making use of it in the last few years, and it's pretty exciting to see them used in more places. My hope is that more scientific and engineering analysis codes will expose derivative interfaces for use in numerical optimization. Here's some short papers with some background for those interested in the topic from a physical engineering perspective: http://www.piercelab.caltech.edu/assets/papers/ftc00.pdf http://www.piercelab.caltech.edu/assets/papers/ftc00.pdf http://www.nt.ntnu.no/users/skoge/prost/proceedings/npcw09/A-N-Ringset-R-1.pdf http://www.nt.ntnu.no/users/skoge/prost/proceedings/npcw09/A...

duvenaud 11y ago
As far as I understand, the adjoint method computes gradients for functions that obey hard constraints (such as fluid solvers), and the main advantage is that it avoids differentiating through iterative constraint satisfaction procedures. I had some success with naively differentiating through fluid solvers, though. Here is a fluid field whose initial velocities have been initialized so that it will end up matching a given image (i.e. blowing a fancy smoke ring:) https://github.com/HIPS/autograd/blob/master/examples/fluidsim/animated.gif https://github.com/HIPS/autograd/blob/master/examples/fluids... and here's a free-form wing shape in the middle of being optimized to maximize lift-to-drag ratio: https://github.com/HIPS/autograd/blob/master/examples/fluidsim/wing.png https://github.com/HIPS/autograd/blob/master/examples/fluids... I too find it baffling when I see engineers doing gradient-free optimization of simulated objective functions, although it's not always easy to compute gradients, especially when using massive Fortran codebases or very large-scale simulations.

by thearn4 11y ago

duvenaud 11y ago

As far as I understand, the adjoint method computes gradients for functions that obey hard constraints (such as fluid solvers), and the main advantage is that it avoids differentiating through iterative constraint satisfaction procedures. I had some success with naively differentiating through fluid solvers, though. Here is a fluid field whose initial velocities have been initialized so that it will end up matching a given image (i.e. blowing a fancy smoke ring:) https://github.com/HIPS/autograd/blob/master/examples/fluidsim/animated.gif https://github.com/HIPS/autograd/blob/master/examples/fluids... and here's a free-form wing shape in the middle of being optimized to maximize lift-to-drag ratio: https://github.com/HIPS/autograd/blob/master/examples/fluidsim/wing.png https://github.com/HIPS/autograd/blob/master/examples/fluids... I too find it baffling when I see engineers doing gradient-free optimization of simulated objective functions, although it's not always easy to compute gradients, especially when using massive Fortran codebases or very large-scale simulations.