5 ms·

AD arbitrary julia code with mutation, closures, built in and user defined functions, fancy data structures etc (given the algorithm itself can be differentiat

by byt143 8y ago

AD arbitrary julia code with mutation, closures, built in and user defined functions, fancy data structures etc (given the algorithm itself can be differentiated).

YetAnotherNick 8y ago
Something like eager execution in tensorflow?
- byt143 8y ago
  Except much faster and much more generic. Eager execution is very limited in the subset of python it can differentiate, for example. There's more info in the OP blog post.
- ChrisRackauckas 8y ago
  Eager execution in TensorFlow doesn't work on all Python code or all Swift code. What I am essentially asking for is equivalent to sticking AD on the source code of say any part of SciPy and expecting it to work without hard coding workarounds. Here's my deal. I wrote and maintain the differential equation solver library in Julia. This thing is a huge piece of code composed of almost 70 packages and took something like 5,000 commits and almost entirely pure-Julia. I will keep writing and maintaining this code as its own research project for many reasons. And, sometimes, I want to AD this code or stick it in a neural network. It's a very non-standard application of this kind of code, so it wasn't built to do this from the start. But I am a greedy hacker. I don't want to have to re-write it onto some computational graph package (TensorFlow) or build interfaces. I want to just call some AD library function on any pure-Julia function in my package and have it output derivatives just like it was any numerical diff package. ForwardDiff.jl, ReverseDiff.jl, Flux.jl, etc. all have autodiffs that work on these routines. I find that magical. In fact, I was shocked that it can be faster than the standard way of calculating these derivatives via sensitivity analysis (we'll put a paper out on Arxiv in a few days about this). There's still a few hiccups that can be fixed up for non-ML applications, but these Julia differentiation tools have really impressed me.
  - ChrisFoster 8y ago
    > faster than the standard way of calculating these derivatives via sensitivity analysis This is quite shocking indeed, I'll look for the paper when it comes out. I'm particularly interested in whether this is true for the sensitivity functions used in shape optimization of composite materials. For example, when optimizing for a stiff, conductive material, eg https://www.sciencedirect.com/science/article/pii/S0020768308000929?via%3Dihub https://www.sciencedirect.com/science/article/pii/S002076830...
    - ChrisRackauckas 8y ago
      This is it: https://arxiv.org/abs/1812.01892 https://arxiv.org/abs/1812.01892
      - ChrisFoster 8y ago
        Thanks! At a quick glance looks like this may not be useful for the problem I had in mind, but I'll have to read it in more detail.

by byt143 8y ago

AD arbitrary julia code with mutation, closures, built in and user defined functions, fancy data structures etc (given the algorithm itself can be differentiated).

YetAnotherNick 8y ago

Something like eager execution in tensorflow?

byt143 8y ago
Except much faster and much more generic. Eager execution is very limited in the subset of python it can differentiate, for example. There's more info in the OP blog post.
ChrisRackauckas 8y ago
Eager execution in TensorFlow doesn't work on all Python code or all Swift code. What I am essentially asking for is equivalent to sticking AD on the source code of say any part of SciPy and expecting it to work without hard coding workarounds. Here's my deal. I wrote and maintain the differential equation solver library in Julia. This thing is a huge piece of code composed of almost 70 packages and took something like 5,000 commits and almost entirely pure-Julia. I will keep writing and maintaining this code as its own research project for many reasons. And, sometimes, I want to AD this code or stick it in a neural network. It's a very non-standard application of this kind of code, so it wasn't built to do this from the start. But I am a greedy hacker. I don't want to have to re-write it onto some computational graph package (TensorFlow) or build interfaces. I want to just call some AD library function on any pure-Julia function in my package and have it output derivatives just like it was any numerical diff package. ForwardDiff.jl, ReverseDiff.jl, Flux.jl, etc. all have autodiffs that work on these routines. I find that magical. In fact, I was shocked that it can be faster than the standard way of calculating these derivatives via sensitivity analysis (we'll put a paper out on Arxiv in a few days about this). There's still a few hiccups that can be fixed up for non-ML applications, but these Julia differentiation tools have really impressed me.
- ChrisFoster 8y ago
  > faster than the standard way of calculating these derivatives via sensitivity analysis This is quite shocking indeed, I'll look for the paper when it comes out. I'm particularly interested in whether this is true for the sensitivity functions used in shape optimization of composite materials. For example, when optimizing for a stiff, conductive material, eg https://www.sciencedirect.com/science/article/pii/S0020768308000929?via%3Dihub https://www.sciencedirect.com/science/article/pii/S002076830...
  - ChrisRackauckas 8y ago
    This is it: https://arxiv.org/abs/1812.01892 https://arxiv.org/abs/1812.01892
    - ChrisFoster 8y ago
      Thanks! At a quick glance looks like this may not be useful for the problem I had in mind, but I'll have to read it in more detail.