4 ms·

As someone who works in merging differential equations and machine learning, I have found this kind of work essential for what I do. Pervasive AD that allows me

As someone who works in merging differential equations and machine learning, I have found this kind of work essential for what I do. Pervasive AD that allows merging neural networks and diffeq solvers is allowing us to explore all of kinds of new models and new problems. Sure it doesn't impact vanilla machine learning all that much (though Zygote.jl does allow for a lot of optimizations that wouldn't be possible with tracing-based AD), but it definitely opens up a new wave of AI possibilities.

YetAnotherNick 8y ago
Can you define what is pervasive automatic differentiation please.
- byt143 8y ago
  AD arbitrary julia code with mutation, closures, built in and user defined functions, fancy data structures etc (given the algorithm itself can be differentiated).
  - YetAnotherNick 8y ago
    Something like eager execution in tensorflow?
    - byt143 8y ago
      Except much faster and much more generic. Eager execution is very limited in the subset of python it can differentiate, for example. There's more info in the OP blog post.
    - ChrisRackauckas 8y ago
      Eager execution in TensorFlow doesn't work on all Python code or all Swift code. What I am essentially asking for is equivalent to sticking AD on the source code of say any part of SciPy and expecting it to work without hard coding workarounds. Here's my deal. I wrote and maintain the differential equation solver library in Julia. This thing is a huge piece of code composed of almost 70 packages and took something like 5,000 commits and almost entirely pure-Julia. I will keep writing and maintaining this code as its own research project for many reasons. And, sometimes, I want to AD this code or stick it in a neural network. It's a very non-standard application of this kind of code, so it wasn't built to do this from the start. But I am a greedy hacker. I don't want to have to re-write it onto some computational graph package (TensorFlow) or build interfaces. I want to just call some AD library function on any pure-Julia function in my package and have it output derivatives just like it was any numerical diff package. ForwardDiff.jl, ReverseDiff.jl, Flux.jl, etc. all have autodiffs that work on these routines. I find that magical. In fact, I was shocked that it can be faster than the standard way of calculating these derivatives via sensitivity analysis (we'll put a paper out on Arxiv in a few days about this). There's still a few hiccups that can be fixed up for non-ML applications, but these Julia differentiation tools have really impressed me.
      - ChrisFoster 8y ago
        > faster than the standard way of calculating these derivatives via sensitivity analysis This is quite shocking indeed, I'll look for the paper when it comes out. I'm particularly interested in whether this is true for the sensitivity functions used in shape optimization of composite materials. For example, when optimizing for a stiff, conductive material, eg https://www.sciencedirect.com/science/article/pii/S0020768308000929?via%3Dihub https://www.sciencedirect.com/science/article/pii/S002076830...
        ChrisRackauckas 8y ago
        This is it: https://arxiv.org/abs/1812.01892 https://arxiv.org/abs/1812.01892
        ChrisFoster 8y ago
        Thanks! At a quick glance looks like this may not be useful for the problem I had in mind, but I'll have to read it in more detail.
S4M 8y ago
> As someone who works in merging differential equations and machine learning That sounds really interesting, could you expand on your work/research?
- faitswulff 8y ago
  A bit of googling yields http://chrisrackauckas.com/ http://chrisrackauckas.com/ which seems like a likely match.
ArtWomb 8y ago
Neural ODE solver just announced as a Best Paper at NeurIPS 2018 conference. What's interesting is that not only do preliminary results yield significantly less error in prediction vs RNNs. Differentiable models allow for extrapolations well beyond range of observable test data set. Neural Ordinary Differential Equations https://arxiv.org/pdf/1806.07366.pdf https://arxiv.org/pdf/1806.07366.pdf
- ChrisRackauckas 8y ago
  I am working on very different approaches with very different applications, but yes the neural ODE (not a solver BTW) is a good example of this kind of application and the effectiveness of the approach. It's a fantastic paper if anyone hasn't read it.

YetAnotherNick 8y ago

Can you define what is pervasive automatic differentiation please.

byt143 8y ago
AD arbitrary julia code with mutation, closures, built in and user defined functions, fancy data structures etc (given the algorithm itself can be differentiated).
- YetAnotherNick 8y ago
  Something like eager execution in tensorflow?
  - byt143 8y ago
    Except much faster and much more generic. Eager execution is very limited in the subset of python it can differentiate, for example. There's more info in the OP blog post.
  - ChrisRackauckas 8y ago
    Eager execution in TensorFlow doesn't work on all Python code or all Swift code. What I am essentially asking for is equivalent to sticking AD on the source code of say any part of SciPy and expecting it to work without hard coding workarounds. Here's my deal. I wrote and maintain the differential equation solver library in Julia. This thing is a huge piece of code composed of almost 70 packages and took something like 5,000 commits and almost entirely pure-Julia. I will keep writing and maintaining this code as its own research project for many reasons. And, sometimes, I want to AD this code or stick it in a neural network. It's a very non-standard application of this kind of code, so it wasn't built to do this from the start. But I am a greedy hacker. I don't want to have to re-write it onto some computational graph package (TensorFlow) or build interfaces. I want to just call some AD library function on any pure-Julia function in my package and have it output derivatives just like it was any numerical diff package. ForwardDiff.jl, ReverseDiff.jl, Flux.jl, etc. all have autodiffs that work on these routines. I find that magical. In fact, I was shocked that it can be faster than the standard way of calculating these derivatives via sensitivity analysis (we'll put a paper out on Arxiv in a few days about this). There's still a few hiccups that can be fixed up for non-ML applications, but these Julia differentiation tools have really impressed me.
    - ChrisFoster 8y ago
      > faster than the standard way of calculating these derivatives via sensitivity analysis This is quite shocking indeed, I'll look for the paper when it comes out. I'm particularly interested in whether this is true for the sensitivity functions used in shape optimization of composite materials. For example, when optimizing for a stiff, conductive material, eg https://www.sciencedirect.com/science/article/pii/S0020768308000929?via%3Dihub https://www.sciencedirect.com/science/article/pii/S002076830...
      - ChrisRackauckas 8y ago
        This is it: https://arxiv.org/abs/1812.01892 https://arxiv.org/abs/1812.01892
        ChrisFoster 8y ago
        Thanks! At a quick glance looks like this may not be useful for the problem I had in mind, but I'll have to read it in more detail.

S4M 8y ago

> As someone who works in merging differential equations and machine learning That sounds really interesting, could you expand on your work/research?

faitswulff 8y ago
A bit of googling yields http://chrisrackauckas.com/ http://chrisrackauckas.com/ which seems like a likely match.

ArtWomb 8y ago

Neural ODE solver just announced as a Best Paper at NeurIPS 2018 conference. What's interesting is that not only do preliminary results yield significantly less error in prediction vs RNNs. Differentiable models allow for extrapolations well beyond range of observable test data set. Neural Ordinary Differential Equations https://arxiv.org/pdf/1806.07366.pdf https://arxiv.org/pdf/1806.07366.pdf

ChrisRackauckas 8y ago
I am working on very different approaches with very different applications, but yes the neural ODE (not a solver BTW) is a good example of this kind of application and the effectiveness of the approach. It's a fantastic paper if anyone hasn't read it.