3 ms·

Since the blog post does not have many code samples, this non-trivial AD example with Zygote.jl is worth sharing (it's from their readme): julia> using Zyg

Since the blog post does not have many code samples, this non-trivial AD example with Zygote.jl is worth sharing (it's from their readme): julia> using Zygote julia> fs = Dict("sin" => sin, "cos" => cos); julia> derivative(x -> fs[readline()](x), 1.0) cos -0.8414709848078965 julia> -sin(1.0) # 'true' derivative in 1.0 -0.8414709848078965 So Zygote can apply AD to an anonymous function that looks up a function in a hash table from user input. https://github.com/FluxML/Zygote.jl https://github.com/FluxML/Zygote.jl

UncleOxidant 8y ago
That's pretty impressive. I notice this in the zygote readme: "The Julia compiler does not yet support all features needed to make Zygote fast, particularly in the presence of control flow." Any idea of when the Julia compiler will support these features?
- viksit 8y ago
  Could you elaborate why this is so impressive? As I understand it, it's because things like TF or PyTorch can only AD specific things as part of a spec vs generic functions like this?
  - twtw 8y ago
    Yeah, it's impressive because it's differentiating the language - it doesn't require putting 'tf.' in front of everything and hoping tf has the function you want.
  - FridgeSeal 8y ago
    It's impressive and super super cool, because now any function you can write in Julia, the AD can differentiate. You don't have to hope that your AD/ML package has that function, you can just write it or find it in a package and punch it straight in. That's awesome.
    - 0-_-0 8y ago
      How is that different from e.g. Tensorflow? You can compose any function you want that has differentiable components, and (re)define the derivatives of elementary or composed functions.
      - Setepenre 8y ago
        No tensorflow does not allow you to do that. Tensorflow differentiate a TF graph. Flux differentiate Julia's IR. Tensorflow has OP(eration)s that are differentiable that you can compose and that's it. If you want to implement something that is going to be differentiable you need to implement it using Tensorflow OPs or add your own OPs to [tensorflow] with their gradient. With Flux you can take code written by a random guy on the internet that never thought about using his stuff in ML and Flux will be able to differentiate it anyway. [1]: https://www.tensorflow.org/guide/extend/op https://www.tensorflow.org/guide/extend/op
        0-_-0 8y ago
        You mean like AutoGraph in Tensorflow? https://github.com/tensorflow/tensorflow/tree/master/tensorflow/python/autograph https://github.com/tensorflow/tensorflow/tree/master/tensorf...
        byt143 8y ago
        Still a very restricted subset of python, can't autodiff custom types etc
        FridgeSeal 8y ago
        Still a small subset of the language and features They've also got to manually implement the Python -> autograph translation for a whole variety of language features (so any language features that get added or changed will break autograph until it's updated. Flux gets this essentially for free, for the entire Julia language, without the need to manually build that language -> tensorflow translation layer. With the added benefit of Julia's non-trivial performance benefits.

UncleOxidant 8y ago

That's pretty impressive. I notice this in the zygote readme: "The Julia compiler does not yet support all features needed to make Zygote fast, particularly in the presence of control flow." Any idea of when the Julia compiler will support these features?

viksit 8y ago
Could you elaborate why this is so impressive? As I understand it, it's because things like TF or PyTorch can only AD specific things as part of a spec vs generic functions like this?
- twtw 8y ago
  Yeah, it's impressive because it's differentiating the language - it doesn't require putting 'tf.' in front of everything and hoping tf has the function you want.
- FridgeSeal 8y ago
  It's impressive and super super cool, because now any function you can write in Julia, the AD can differentiate. You don't have to hope that your AD/ML package has that function, you can just write it or find it in a package and punch it straight in. That's awesome.
  - 0-_-0 8y ago
    How is that different from e.g. Tensorflow? You can compose any function you want that has differentiable components, and (re)define the derivatives of elementary or composed functions.
    - Setepenre 8y ago
      No tensorflow does not allow you to do that. Tensorflow differentiate a TF graph. Flux differentiate Julia's IR. Tensorflow has OP(eration)s that are differentiable that you can compose and that's it. If you want to implement something that is going to be differentiable you need to implement it using Tensorflow OPs or add your own OPs to [tensorflow] with their gradient. With Flux you can take code written by a random guy on the internet that never thought about using his stuff in ML and Flux will be able to differentiate it anyway. [1]: https://www.tensorflow.org/guide/extend/op https://www.tensorflow.org/guide/extend/op
      - 0-_-0 8y ago
        You mean like AutoGraph in Tensorflow? https://github.com/tensorflow/tensorflow/tree/master/tensorflow/python/autograph https://github.com/tensorflow/tensorflow/tree/master/tensorf...
        byt143 8y ago
        Still a very restricted subset of python, can't autodiff custom types etc
        FridgeSeal 8y ago
        Still a small subset of the language and features They've also got to manually implement the Python -> autograph translation for a whole variety of language features (so any language features that get added or changed will break autograph until it's updated. Flux gets this essentially for free, for the entire Julia language, without the need to manually build that language -> tensorflow translation layer. With the added benefit of Julia's non-trivial performance benefits.