7 ms·

Yeah I agree with Lyndon, Zygote is a much more ambitious project than Jax, so I would not be surprised if we see them make quicker progress than Zygote in this

Yeah I agree with Lyndon, Zygote is a much more ambitious project than Jax, so I would not be surprised if we see them make quicker progress than Zygote in this space.

UncleOxidant 7y ago
I wonder if the Zygote folks would be better off focusing on a subset of Julia for now instead of supporting all of Julia. And then focusing on performance for that subset. Maybe this is more difficult because of the way Zygote is implemented (via Julia's macro system) which just lends itself to whole-language support?
- eigenspace 7y ago
  It's hard to say, I really do think that what the Zygote folks are trying to do is quite achievable, but I also worry that Zygote's prominence may have been bad for the Julia autodiff community. A year or two ago, it at least seemed to be a much more vibrant situation where there was more effort going into lots of different approaches, but then Zygote really picked up steam and it seemed like it was going to solve all our problems and cure cancer so development on the other packages slowed down, but now the 'finish line' for Zygote doesn't seem to be as close as we initially thought.
  - UncleOxidant 7y ago
    On the other hand, there are a lot less Julia developers than Python developers so focusing on Zygote instead of splitting the effort over several AD packages makes some sense, doesn't it? There were already a few AD packages that were fairly mature: https://www.juliadiff.org/ https://www.juliadiff.org/ Also, I think performance is very important because Zygote is used in Flux (Julia's main deep learning framework at this point.) and if it's keeping Flux's performance from matching, say, PyTorch's then that's going to limit adoption of Flux and Julia. One of Julia's main claims to fame is performance so people coming to kick the tires are going to be disappointed if it's actually slower for this domain.
    - eigenspace 7y ago
      I think my concern is that the main Zygote dev is not very responsive or communicative. That's fine, he doesn't owe the rest of the community anything, but it also seems irresponsible to put all our eggs in his basket (note I'm by no means knowledgeable about this and I'm open to the idea that it's a mis-characterization. This is merely my impression as a relative outsider looking in). Besides, any implementation of AD is going to have downsides. Having many different, hot swappable implementations is quite nice because you can tailor towards your needs.
    - oxinabox 7y ago
      We have so many AD packages than that in julia. Julia suffers the lisp problem for AD. http://winestockwebdesign.com/Essays/Lisp_Curse.html http://winestockwebdesign.com/Essays/Lisp_Curse.html It is a real problem. One of my on going projects is ChainRules (http://www.juliadiff.org/ChainRulesCore.jl/dev/ http://www.juliadiff.org/ChainRulesCore.jl/dev/) which will unite them under one set of custom senstitivities and more generally make it easier to mix and match them I need to update the JuliaDiff website, I want to list all of them in a table with some some key points. Forward mode: - [ForwardDiff](https://github.com/JuliaDiff/ForwardDiff.jl https://github.com/JuliaDiff/ForwardDiff.jl) - [ForwardDiff2](https://github.com/YingboMa//ForwardDiff2.jl https://github.com/YingboMa//ForwardDiff2.jl) Reverse Mode: - [Nabla](https://github.com/invenia/Nabla.jl/ https://github.com/invenia/Nabla.jl/) - [Tracker](https://github.com/FluxML/Tracker.jl https://github.com/FluxML/Tracker.jl) - [Yota](https://github.com/dfdx/Yota.jl https://github.com/dfdx/Yota.jl) - [Zygote](https://github.com/FluxML/Zygote.jl https://github.com/FluxML/Zygote.jl) - [ReverseDiff](https://github.com/JuliaDiff/ReverseDiff.jl https://github.com/JuliaDiff/ReverseDiff.jl) - [AutoGrad.jl](https://github.com/denizyuret/AutoGrad.jl https://github.com/denizyuret/AutoGrad.jl) - [NiLang](https://github.com/GiggleLiu/NiLang.jl https://github.com/GiggleLiu/NiLang.jl) (arguably not reverse mode) Symbolic: - [ModelingToolKit](https://github.com/JuliaDiffEq/ModelingToolkit.jl https://github.com/JuliaDiffEq/ModelingToolkit.jl) - [XGrad.jl](https://github.com/dfdx/XGrad.jl https://github.com/dfdx/XGrad.jl) Finite Differencing: - [Calculus](https://github.com/JuliaMath/Calculus.jl https://github.com/JuliaMath/Calculus.jl) (please stop) - [FiniteDifferences](https://github.com/JuliaDiff/FiniteDifferences.jl https://github.com/JuliaDiff/FiniteDifferences.jl) - [FiniteDiff](https://github.com/JuliaDiff/FiniteDiff.jl https://github.com/JuliaDiff/FiniteDiff.jl)
- ChrisRackauckas 7y ago
  Maybe. The nice thing with Julia is that all of the ADs just target Julia AST, so you can mix and match them. I'm using ForwardDiff + ReverseDiff compiled tapes with Zygote over it, because each have different optimizations that can be done to different parts of the code. ChainRules.jl is a great package that will allow this to pick up more steam. While I don't expect most users to go this deep into AD, if you actually want to use it in the backend of packages in a very deep and high performance way, there's nothing better than mixing the strengths of different approaches together.

Yeah I agree with Lyndon, Zygote is a much more ambitious project than Jax, so I would not be surprised if we see them make quicker progress than Zygote in this space.

UncleOxidant 7y ago

I wonder if the Zygote folks would be better off focusing on a subset of Julia for now instead of supporting all of Julia. And then focusing on performance for that subset. Maybe this is more difficult because of the way Zygote is implemented (via Julia's macro system) which just lends itself to whole-language support?

eigenspace 7y ago
It's hard to say, I really do think that what the Zygote folks are trying to do is quite achievable, but I also worry that Zygote's prominence may have been bad for the Julia autodiff community. A year or two ago, it at least seemed to be a much more vibrant situation where there was more effort going into lots of different approaches, but then Zygote really picked up steam and it seemed like it was going to solve all our problems and cure cancer so development on the other packages slowed down, but now the 'finish line' for Zygote doesn't seem to be as close as we initially thought.
- UncleOxidant 7y ago
  On the other hand, there are a lot less Julia developers than Python developers so focusing on Zygote instead of splitting the effort over several AD packages makes some sense, doesn't it? There were already a few AD packages that were fairly mature: https://www.juliadiff.org/ https://www.juliadiff.org/ Also, I think performance is very important because Zygote is used in Flux (Julia's main deep learning framework at this point.) and if it's keeping Flux's performance from matching, say, PyTorch's then that's going to limit adoption of Flux and Julia. One of Julia's main claims to fame is performance so people coming to kick the tires are going to be disappointed if it's actually slower for this domain.
  - eigenspace 7y ago
    I think my concern is that the main Zygote dev is not very responsive or communicative. That's fine, he doesn't owe the rest of the community anything, but it also seems irresponsible to put all our eggs in his basket (note I'm by no means knowledgeable about this and I'm open to the idea that it's a mis-characterization. This is merely my impression as a relative outsider looking in). Besides, any implementation of AD is going to have downsides. Having many different, hot swappable implementations is quite nice because you can tailor towards your needs.
  - oxinabox 7y ago
    We have so many AD packages than that in julia. Julia suffers the lisp problem for AD. http://winestockwebdesign.com/Essays/Lisp_Curse.html http://winestockwebdesign.com/Essays/Lisp_Curse.html It is a real problem. One of my on going projects is ChainRules (http://www.juliadiff.org/ChainRulesCore.jl/dev/ http://www.juliadiff.org/ChainRulesCore.jl/dev/) which will unite them under one set of custom senstitivities and more generally make it easier to mix and match them I need to update the JuliaDiff website, I want to list all of them in a table with some some key points. Forward mode: - [ForwardDiff](https://github.com/JuliaDiff/ForwardDiff.jl https://github.com/JuliaDiff/ForwardDiff.jl) - [ForwardDiff2](https://github.com/YingboMa//ForwardDiff2.jl https://github.com/YingboMa//ForwardDiff2.jl) Reverse Mode: - [Nabla](https://github.com/invenia/Nabla.jl/ https://github.com/invenia/Nabla.jl/) - [Tracker](https://github.com/FluxML/Tracker.jl https://github.com/FluxML/Tracker.jl) - [Yota](https://github.com/dfdx/Yota.jl https://github.com/dfdx/Yota.jl) - [Zygote](https://github.com/FluxML/Zygote.jl https://github.com/FluxML/Zygote.jl) - [ReverseDiff](https://github.com/JuliaDiff/ReverseDiff.jl https://github.com/JuliaDiff/ReverseDiff.jl) - [AutoGrad.jl](https://github.com/denizyuret/AutoGrad.jl https://github.com/denizyuret/AutoGrad.jl) - [NiLang](https://github.com/GiggleLiu/NiLang.jl https://github.com/GiggleLiu/NiLang.jl) (arguably not reverse mode) Symbolic: - [ModelingToolKit](https://github.com/JuliaDiffEq/ModelingToolkit.jl https://github.com/JuliaDiffEq/ModelingToolkit.jl) - [XGrad.jl](https://github.com/dfdx/XGrad.jl https://github.com/dfdx/XGrad.jl) Finite Differencing: - [Calculus](https://github.com/JuliaMath/Calculus.jl https://github.com/JuliaMath/Calculus.jl) (please stop) - [FiniteDifferences](https://github.com/JuliaDiff/FiniteDifferences.jl https://github.com/JuliaDiff/FiniteDifferences.jl) - [FiniteDiff](https://github.com/JuliaDiff/FiniteDiff.jl https://github.com/JuliaDiff/FiniteDiff.jl)
ChrisRackauckas 7y ago
Maybe. The nice thing with Julia is that all of the ADs just target Julia AST, so you can mix and match them. I'm using ForwardDiff + ReverseDiff compiled tapes with Zygote over it, because each have different optimizations that can be done to different parts of the code. ChainRules.jl is a great package that will allow this to pick up more steam. While I don't expect most users to go this deep into AD, if you actually want to use it in the backend of packages in a very deep and high performance way, there's nothing better than mixing the strengths of different approaches together.