3 ms·

It's hard to say, I really do think that what the Zygote folks are trying to do is quite achievable, but I also worry that Zygote's prominence may have been bad

It's hard to say, I really do think that what the Zygote folks are trying to do is quite achievable, but I also worry that Zygote's prominence may have been bad for the Julia autodiff community. A year or two ago, it at least seemed to be a much more vibrant situation where there was more effort going into lots of different approaches, but then Zygote really picked up steam and it seemed like it was going to solve all our problems and cure cancer so development on the other packages slowed down, but now the 'finish line' for Zygote doesn't seem to be as close as we initially thought.

UncleOxidant 7y ago
On the other hand, there are a lot less Julia developers than Python developers so focusing on Zygote instead of splitting the effort over several AD packages makes some sense, doesn't it? There were already a few AD packages that were fairly mature: https://www.juliadiff.org/ https://www.juliadiff.org/ Also, I think performance is very important because Zygote is used in Flux (Julia's main deep learning framework at this point.) and if it's keeping Flux's performance from matching, say, PyTorch's then that's going to limit adoption of Flux and Julia. One of Julia's main claims to fame is performance so people coming to kick the tires are going to be disappointed if it's actually slower for this domain.
- eigenspace 7y ago
  I think my concern is that the main Zygote dev is not very responsive or communicative. That's fine, he doesn't owe the rest of the community anything, but it also seems irresponsible to put all our eggs in his basket (note I'm by no means knowledgeable about this and I'm open to the idea that it's a mis-characterization. This is merely my impression as a relative outsider looking in). Besides, any implementation of AD is going to have downsides. Having many different, hot swappable implementations is quite nice because you can tailor towards your needs.
- oxinabox 7y ago
  We have so many AD packages than that in julia. Julia suffers the lisp problem for AD. http://winestockwebdesign.com/Essays/Lisp_Curse.html http://winestockwebdesign.com/Essays/Lisp_Curse.html It is a real problem. One of my on going projects is ChainRules (http://www.juliadiff.org/ChainRulesCore.jl/dev/ http://www.juliadiff.org/ChainRulesCore.jl/dev/) which will unite them under one set of custom senstitivities and more generally make it easier to mix and match them I need to update the JuliaDiff website, I want to list all of them in a table with some some key points. Forward mode: - [ForwardDiff](https://github.com/JuliaDiff/ForwardDiff.jl https://github.com/JuliaDiff/ForwardDiff.jl) - [ForwardDiff2](https://github.com/YingboMa//ForwardDiff2.jl https://github.com/YingboMa//ForwardDiff2.jl) Reverse Mode: - [Nabla](https://github.com/invenia/Nabla.jl/ https://github.com/invenia/Nabla.jl/) - [Tracker](https://github.com/FluxML/Tracker.jl https://github.com/FluxML/Tracker.jl) - [Yota](https://github.com/dfdx/Yota.jl https://github.com/dfdx/Yota.jl) - [Zygote](https://github.com/FluxML/Zygote.jl https://github.com/FluxML/Zygote.jl) - [ReverseDiff](https://github.com/JuliaDiff/ReverseDiff.jl https://github.com/JuliaDiff/ReverseDiff.jl) - [AutoGrad.jl](https://github.com/denizyuret/AutoGrad.jl https://github.com/denizyuret/AutoGrad.jl) - [NiLang](https://github.com/GiggleLiu/NiLang.jl https://github.com/GiggleLiu/NiLang.jl) (arguably not reverse mode) Symbolic: - [ModelingToolKit](https://github.com/JuliaDiffEq/ModelingToolkit.jl https://github.com/JuliaDiffEq/ModelingToolkit.jl) - [XGrad.jl](https://github.com/dfdx/XGrad.jl https://github.com/dfdx/XGrad.jl) Finite Differencing: - [Calculus](https://github.com/JuliaMath/Calculus.jl https://github.com/JuliaMath/Calculus.jl) (please stop) - [FiniteDifferences](https://github.com/JuliaDiff/FiniteDifferences.jl https://github.com/JuliaDiff/FiniteDifferences.jl) - [FiniteDiff](https://github.com/JuliaDiff/FiniteDiff.jl https://github.com/JuliaDiff/FiniteDiff.jl)

UncleOxidant 7y ago

On the other hand, there are a lot less Julia developers than Python developers so focusing on Zygote instead of splitting the effort over several AD packages makes some sense, doesn't it? There were already a few AD packages that were fairly mature: https://www.juliadiff.org/ https://www.juliadiff.org/ Also, I think performance is very important because Zygote is used in Flux (Julia's main deep learning framework at this point.) and if it's keeping Flux's performance from matching, say, PyTorch's then that's going to limit adoption of Flux and Julia. One of Julia's main claims to fame is performance so people coming to kick the tires are going to be disappointed if it's actually slower for this domain.

eigenspace 7y ago
I think my concern is that the main Zygote dev is not very responsive or communicative. That's fine, he doesn't owe the rest of the community anything, but it also seems irresponsible to put all our eggs in his basket (note I'm by no means knowledgeable about this and I'm open to the idea that it's a mis-characterization. This is merely my impression as a relative outsider looking in). Besides, any implementation of AD is going to have downsides. Having many different, hot swappable implementations is quite nice because you can tailor towards your needs.
oxinabox 7y ago
We have so many AD packages than that in julia. Julia suffers the lisp problem for AD. http://winestockwebdesign.com/Essays/Lisp_Curse.html http://winestockwebdesign.com/Essays/Lisp_Curse.html It is a real problem. One of my on going projects is ChainRules (http://www.juliadiff.org/ChainRulesCore.jl/dev/ http://www.juliadiff.org/ChainRulesCore.jl/dev/) which will unite them under one set of custom senstitivities and more generally make it easier to mix and match them I need to update the JuliaDiff website, I want to list all of them in a table with some some key points. Forward mode: - [ForwardDiff](https://github.com/JuliaDiff/ForwardDiff.jl https://github.com/JuliaDiff/ForwardDiff.jl) - [ForwardDiff2](https://github.com/YingboMa//ForwardDiff2.jl https://github.com/YingboMa//ForwardDiff2.jl) Reverse Mode: - [Nabla](https://github.com/invenia/Nabla.jl/ https://github.com/invenia/Nabla.jl/) - [Tracker](https://github.com/FluxML/Tracker.jl https://github.com/FluxML/Tracker.jl) - [Yota](https://github.com/dfdx/Yota.jl https://github.com/dfdx/Yota.jl) - [Zygote](https://github.com/FluxML/Zygote.jl https://github.com/FluxML/Zygote.jl) - [ReverseDiff](https://github.com/JuliaDiff/ReverseDiff.jl https://github.com/JuliaDiff/ReverseDiff.jl) - [AutoGrad.jl](https://github.com/denizyuret/AutoGrad.jl https://github.com/denizyuret/AutoGrad.jl) - [NiLang](https://github.com/GiggleLiu/NiLang.jl https://github.com/GiggleLiu/NiLang.jl) (arguably not reverse mode) Symbolic: - [ModelingToolKit](https://github.com/JuliaDiffEq/ModelingToolkit.jl https://github.com/JuliaDiffEq/ModelingToolkit.jl) - [XGrad.jl](https://github.com/dfdx/XGrad.jl https://github.com/dfdx/XGrad.jl) Finite Differencing: - [Calculus](https://github.com/JuliaMath/Calculus.jl https://github.com/JuliaMath/Calculus.jl) (please stop) - [FiniteDifferences](https://github.com/JuliaDiff/FiniteDifferences.jl https://github.com/JuliaDiff/FiniteDifferences.jl) - [FiniteDiff](https://github.com/JuliaDiff/FiniteDiff.jl https://github.com/JuliaDiff/FiniteDiff.jl)