5 ms·

Even though this posts thesis is “trade offs”, it doesn’t really talk about any technical advantages that the Python’s AD ecosystem (Tensorflow, PyTorch, JAX) h

Even though this posts thesis is “trade offs”, it doesn’t really talk about any technical advantages that the Python’s AD ecosystem (Tensorflow, PyTorch, JAX) has over Julia’s (Zygote.jl, Diffractor.jl).

adgjlsfhk1 5y ago
It touches on the main one which is simplicity. it's much easier to write an AD system for a more static language.
civilized 5y ago
Maybe it has no technical advantages, unless you count being very popular and in an accessible language as a technical advantage (which it definitely could be depending on your definition of "technical"). Julia is designed for advanced numerical computing and Python isn't. The metaprogramming affordances needed for AD are much better developed in Julia than they ever will be in Python. And let's not forget the immense utility of multiple dispatch in Julia, another feature Python will probably never have. So it's not surprising that Julia is simply way more capable.
- jaggirs 5y ago
  One disatvantage of the language itself is the need for compilation, which isn't that fast in my limited experience. But I would love to hear how much this affects iteration speed.
  - civilized 5y ago
    Yeah, I'd imagine that for things both Python and Julia can do AD-wise, Python may be preferable since it's interpreted and thus instant-feedback, but all the numerical heavy lifting in packages like Jax and PyTorch is done in fast C++. So you should be getting a more appealing environment for experimentation without losing out on speed.
  - calaphos 5y ago
    The same issue exists with Jax. XLA compilation can take up quite a bit of time, especially on larger NN models. And theres no persistent compile cache, so even if you don't change the jitted function you need to wait for compilation again as you restart the process.
    - alevskaya 5y ago
      Jax does actually already support a persistent compilation cache for TPU, and support for caching GPU compiles is being worked on currently.
- mountainriver 5y ago
  The Julia crowd touting multiple dispatch all the time is so strange, it’s actually one of the main reasons the language hasn’t had much uptake from what I can tell. Python is just more approachable and natural to people. Julia should learn from that
  - civilized 5y ago
    Python is just more OO style, so people who have been taught OOP in school are comfortable with it. That will include the vast majority of generic SWEs writing generic CRUD apps. But personally I find OOP ugly and unnatural, and Julia's model elegant and natural. And far more powerful - Julia programmers are using multiple dispatch to build out scientific computing to a sophistication not seen in any other language. It might not be your cup of tea if you need to see object.method() in your code, but if you're more mentally flexible and want to build the next generation of technical computing tools, Julia is the place to be right now.
    - mountainriver 5y ago
      Yeah I’m definitely mentally flexible and have coded in many paradigms, I don’t love OO and generally don’t write that way but multiple dispatch as a primary design pattern is odd. I’ve tried it for close to a year and the ergonomics still felt off, it reminds me of how the scala crowd talked about functional programming, and we’ve seen how that turned out. I hear this from a lot of people that try Julia and yet the Julia crowds answer is always that they are dumb. Sounds a lot like the scala crowd…
      - civilized 5y ago
        I think 90% of the ergonomics issue is that people want dot notation and tab-autocomplete in their IDE so they can type obj.<tab> and get the methods that operate on obj. Which I agree, some version of that should exist, and there's no real reason it can't exist in Julia. The tooling is just not as mature as other languages. Julia is far ahead in affordances to write fancy technical code and fairly behind in simple things, like standard affordances to write more ordinary code, or the ability to quickly load in data and make a plot. I just think it's a misdiagnosis to blame multiple dispatch for this issue. It's much more about the Julia community prioritizing the needs of their target market.

adgjlsfhk1 5y ago

It touches on the main one which is simplicity. it's much easier to write an AD system for a more static language.

civilized 5y ago

Maybe it has no technical advantages, unless you count being very popular and in an accessible language as a technical advantage (which it definitely could be depending on your definition of "technical"). Julia is designed for advanced numerical computing and Python isn't. The metaprogramming affordances needed for AD are much better developed in Julia than they ever will be in Python. And let's not forget the immense utility of multiple dispatch in Julia, another feature Python will probably never have. So it's not surprising that Julia is simply way more capable.

jaggirs 5y ago
One disatvantage of the language itself is the need for compilation, which isn't that fast in my limited experience. But I would love to hear how much this affects iteration speed.
- civilized 5y ago
  Yeah, I'd imagine that for things both Python and Julia can do AD-wise, Python may be preferable since it's interpreted and thus instant-feedback, but all the numerical heavy lifting in packages like Jax and PyTorch is done in fast C++. So you should be getting a more appealing environment for experimentation without losing out on speed.
- calaphos 5y ago
  The same issue exists with Jax. XLA compilation can take up quite a bit of time, especially on larger NN models. And theres no persistent compile cache, so even if you don't change the jitted function you need to wait for compilation again as you restart the process.
  - alevskaya 5y ago
    Jax does actually already support a persistent compilation cache for TPU, and support for caching GPU compiles is being worked on currently.
mountainriver 5y ago
The Julia crowd touting multiple dispatch all the time is so strange, it’s actually one of the main reasons the language hasn’t had much uptake from what I can tell. Python is just more approachable and natural to people. Julia should learn from that
- civilized 5y ago
  Python is just more OO style, so people who have been taught OOP in school are comfortable with it. That will include the vast majority of generic SWEs writing generic CRUD apps. But personally I find OOP ugly and unnatural, and Julia's model elegant and natural. And far more powerful - Julia programmers are using multiple dispatch to build out scientific computing to a sophistication not seen in any other language. It might not be your cup of tea if you need to see object.method() in your code, but if you're more mentally flexible and want to build the next generation of technical computing tools, Julia is the place to be right now.
  - mountainriver 5y ago
    Yeah I’m definitely mentally flexible and have coded in many paradigms, I don’t love OO and generally don’t write that way but multiple dispatch as a primary design pattern is odd. I’ve tried it for close to a year and the ergonomics still felt off, it reminds me of how the scala crowd talked about functional programming, and we’ve seen how that turned out. I hear this from a lot of people that try Julia and yet the Julia crowds answer is always that they are dumb. Sounds a lot like the scala crowd…
    - civilized 5y ago
      I think 90% of the ergonomics issue is that people want dot notation and tab-autocomplete in their IDE so they can type obj.<tab> and get the methods that operate on obj. Which I agree, some version of that should exist, and there's no real reason it can't exist in Julia. The tooling is just not as mature as other languages. Julia is far ahead in affordances to write fancy technical code and fairly behind in simple things, like standard affordances to write more ordinary code, or the ability to quickly load in data and make a plot. I just think it's a misdiagnosis to blame multiple dispatch for this issue. It's much more about the Julia community prioritizing the needs of their target market.