4 ms·

Julia is next gen. The whole experience is so much more pleasant than anything in the space. The interop is also crazy. How is it possible that you can call Mat

Julia is next gen. The whole experience is so much more pleasant than anything in the space. The interop is also crazy. How is it possible that you can call Matlab, Python, cpp, fortran, Rust from one language? I legit think that even if you are using say pytorch. You are better off writing your code in julia and using the python interop.

amkkma 6y ago
Using pytorch/keras/TF from Julia is possible, and even pleasant: https://twitter.com/aureliengeron/status/1277751121440698368 https://twitter.com/aureliengeron/status/1277751121440698368 Though I much prefer Flux.jl
- UncleOxidant 6y ago
  In my experience Flux.jl has a ways to go before it reaches Pytorch performance. Apparently this has something to do with some very efficient GPU kernels that Pytorch uses - I wonder if it's possible for Flux to "borrow" those GPU kernels? It's possible to write GPU kernels in Julia so hopefully this will help close the gap.
  - amkkma 6y ago
    The short answer is yes, it's possible and easy. The longer answer is, that's a bit of an apples to oranges comparison. Pytorch is calling optimized c++ kernels, whereas Flux is completely generic and can do GPU and CPU codegen (including AMD). Flux can do things Pytorch just doesn't have CUDA code for, It's more generic and composable and it's getting there with a much smaller team/budget. That means it has more potential, with the drawback that it isn't quite as fast, for now, in some of the common use cases the pytorch has been hard coded/ hyperoptimized for. In other more complex cases beyond big fat matmuls and such, it's much faster, even comparing to torchscript (sciml, neural ODEs etc). Still, the codegen is getting there with the funded work on the GPU compiler and memory allocation and there eventually won't be any sort of tradeoff. In the meantime, Yes, there is a package at allows calling pytorch kernels without modifying your code. This is also only possible with such finesse due to Julia's power and composability. See here for the package: https://fluxml.ai/2020/06/29/acclerating-flux-torch.html https://fluxml.ai/2020/06/29/acclerating-flux-torch.html
  - ViralBShah 6y ago
    Flux already has the ability to borrow Torch kernels. This was released fairly recently. https://github.com/FluxML/Torch.jl https://github.com/FluxML/Torch.jl https://fluxml.ai/2020/06/29/acclerating-flux-torch.html https://fluxml.ai/2020/06/29/acclerating-flux-torch.html Also the gap between native GPU codegen in Julia and the hand-optimized kernels in PyTorch is lesser than you'd imagine. In fact, see this talk at JuliaCon, which compares Julia on a 1000 GPUs against a CUDA/C implementation, and finds them to be extremely close. https://www.youtube.com/watch?v=vPsfZUqI4_0 https://www.youtube.com/watch?v=vPsfZUqI4_0 https://github.com/eth-cscs/ImplicitGlobalGrid.jl https://github.com/eth-cscs/ImplicitGlobalGrid.jl
    - UncleOxidant 6y ago
      Is this just wrapping the torch kernels or the entire torch API? I was under the impression it was the latter.
  - eigenspace 6y ago
    I feel I should point out that if you need performance and you're mostly only interested in classic PyTorch or TensorFlow style machine learning (i.e. you don't need Flux.jl's characteristic flexibility), then you should really check out KNet.jl [1]. KNet is generally around the same flexibility and performance as something like PyTorch. It doesn't get nearly as much attention as Flux because it's less novel, but it really should be looked at more. [1] https://github.com/denizyuret/Knet.jl https://github.com/denizyuret/Knet.jl
    - ViralBShah 6y ago
      Knet.jl is pretty awesome. The main challenge has been compiling its cuda kernel library. Deniz has helped anyone who ran into issues. However, we are integrating a lot of the underlying infrastructure to the point that both Knet and Flux will only differ in API approach.
  - ChrisRackauckas 6y ago
    It really depends on where you're looking. We tested neural ODE compatible solver benchmarks from 3 ODEs and about 1 million ODEs and saw an advantage for Julia of >30x across the board [1]. Then showed the spiral neural ODE from the original paper train 100x faster in Julia [2]. So it should at least be qualified as, PyTorch has some good kernels, and it has some things it doesn't optimize close to as much as Julia. [1] https://gist.github.com/ChrisRackauckas/cc6ac746e2dfd285c28e0584a2bfd320 https://gist.github.com/ChrisRackauckas/cc6ac746e2dfd285c28e... [2] https://gist.github.com/ChrisRackauckas/4a4d526c15cc4170ce37da837bfc32c4 https://gist.github.com/ChrisRackauckas/4a4d526c15cc4170ce37...
smabie 6y ago
Yeah, the quality of the language is pretty incredible. Especially compared to Python, which has been a disaster for as long as I can remember. I think the ecosystem could be a bit better, but that's more of a personal opinion rather than a serious concern as, like you alluded to, you the interop story is insanely good.
- adamnemecek 6y ago
  The ecosystem is moving very rapidly though. What do you think is missing?
  - smabie 6y ago
    Right now, the DataFrame package doesn't support row indices like Pandas does. Row indices are especially useful time series data, and there's another package called TimeSeries that support a dataframe-like type called a TimeArray that supports dates as row indices. TimeArray is like a DataFrame, but with a lot less features. You can't really do that much with them, and they only have limited support in an auxiliary package for resampling. It would be nice if a new DataFrame library could be created that was inspired by Pandas. In general, I think Julia suffers from the problem of too many small packages. For example, there's 3 libraries I have to use for time series: TimeSeries.jl, TimeSeries Resample.jl,and TimeSeriesIO.jl. None of the libraries are particularly large and should all be merged into one package. Part of the reason for the dirth of small packages is because, unlike Python, Julia's strong support for function polymorphism allows for seamless extension of existing functionality with new packages. Unfortunately, I think this has created a cultural problem in which new packages are created that only slightly modify existing functionality.
    - longemen3000 6y ago
      One of the discussions on Juliacon was exactly this! How to index various things. This was one of the documents produced during the discussion : https://docs.google.com/document/d/1imBX3k0EEejauWVyXONZDRj8LTr0PeLOJNGEgo6ow1g/edit?usp=drivesdk https://docs.google.com/document/d/1imBX3k0EEejauWVyXONZDRj8...

amkkma 6y ago

Using pytorch/keras/TF from Julia is possible, and even pleasant: https://twitter.com/aureliengeron/status/1277751121440698368 https://twitter.com/aureliengeron/status/1277751121440698368 Though I much prefer Flux.jl

UncleOxidant 6y ago
In my experience Flux.jl has a ways to go before it reaches Pytorch performance. Apparently this has something to do with some very efficient GPU kernels that Pytorch uses - I wonder if it's possible for Flux to "borrow" those GPU kernels? It's possible to write GPU kernels in Julia so hopefully this will help close the gap.
- amkkma 6y ago
  The short answer is yes, it's possible and easy. The longer answer is, that's a bit of an apples to oranges comparison. Pytorch is calling optimized c++ kernels, whereas Flux is completely generic and can do GPU and CPU codegen (including AMD). Flux can do things Pytorch just doesn't have CUDA code for, It's more generic and composable and it's getting there with a much smaller team/budget. That means it has more potential, with the drawback that it isn't quite as fast, for now, in some of the common use cases the pytorch has been hard coded/ hyperoptimized for. In other more complex cases beyond big fat matmuls and such, it's much faster, even comparing to torchscript (sciml, neural ODEs etc). Still, the codegen is getting there with the funded work on the GPU compiler and memory allocation and there eventually won't be any sort of tradeoff. In the meantime, Yes, there is a package at allows calling pytorch kernels without modifying your code. This is also only possible with such finesse due to Julia's power and composability. See here for the package: https://fluxml.ai/2020/06/29/acclerating-flux-torch.html https://fluxml.ai/2020/06/29/acclerating-flux-torch.html
- ViralBShah 6y ago
  Flux already has the ability to borrow Torch kernels. This was released fairly recently. https://github.com/FluxML/Torch.jl https://github.com/FluxML/Torch.jl https://fluxml.ai/2020/06/29/acclerating-flux-torch.html https://fluxml.ai/2020/06/29/acclerating-flux-torch.html Also the gap between native GPU codegen in Julia and the hand-optimized kernels in PyTorch is lesser than you'd imagine. In fact, see this talk at JuliaCon, which compares Julia on a 1000 GPUs against a CUDA/C implementation, and finds them to be extremely close. https://www.youtube.com/watch?v=vPsfZUqI4_0 https://www.youtube.com/watch?v=vPsfZUqI4_0 https://github.com/eth-cscs/ImplicitGlobalGrid.jl https://github.com/eth-cscs/ImplicitGlobalGrid.jl
  - UncleOxidant 6y ago
    Is this just wrapping the torch kernels or the entire torch API? I was under the impression it was the latter.
- eigenspace 6y ago
  I feel I should point out that if you need performance and you're mostly only interested in classic PyTorch or TensorFlow style machine learning (i.e. you don't need Flux.jl's characteristic flexibility), then you should really check out KNet.jl [1]. KNet is generally around the same flexibility and performance as something like PyTorch. It doesn't get nearly as much attention as Flux because it's less novel, but it really should be looked at more. [1] https://github.com/denizyuret/Knet.jl https://github.com/denizyuret/Knet.jl
  - ViralBShah 6y ago
    Knet.jl is pretty awesome. The main challenge has been compiling its cuda kernel library. Deniz has helped anyone who ran into issues. However, we are integrating a lot of the underlying infrastructure to the point that both Knet and Flux will only differ in API approach.
- ChrisRackauckas 6y ago
  It really depends on where you're looking. We tested neural ODE compatible solver benchmarks from 3 ODEs and about 1 million ODEs and saw an advantage for Julia of >30x across the board [1]. Then showed the spiral neural ODE from the original paper train 100x faster in Julia [2]. So it should at least be qualified as, PyTorch has some good kernels, and it has some things it doesn't optimize close to as much as Julia. [1] https://gist.github.com/ChrisRackauckas/cc6ac746e2dfd285c28e0584a2bfd320 https://gist.github.com/ChrisRackauckas/cc6ac746e2dfd285c28e... [2] https://gist.github.com/ChrisRackauckas/4a4d526c15cc4170ce37da837bfc32c4 https://gist.github.com/ChrisRackauckas/4a4d526c15cc4170ce37...

smabie 6y ago

Yeah, the quality of the language is pretty incredible. Especially compared to Python, which has been a disaster for as long as I can remember. I think the ecosystem could be a bit better, but that's more of a personal opinion rather than a serious concern as, like you alluded to, you the interop story is insanely good.

adamnemecek 6y ago
The ecosystem is moving very rapidly though. What do you think is missing?
- smabie 6y ago
  Right now, the DataFrame package doesn't support row indices like Pandas does. Row indices are especially useful time series data, and there's another package called TimeSeries that support a dataframe-like type called a TimeArray that supports dates as row indices. TimeArray is like a DataFrame, but with a lot less features. You can't really do that much with them, and they only have limited support in an auxiliary package for resampling. It would be nice if a new DataFrame library could be created that was inspired by Pandas. In general, I think Julia suffers from the problem of too many small packages. For example, there's 3 libraries I have to use for time series: TimeSeries.jl, TimeSeries Resample.jl,and TimeSeriesIO.jl. None of the libraries are particularly large and should all be merged into one package. Part of the reason for the dirth of small packages is because, unlike Python, Julia's strong support for function polymorphism allows for seamless extension of existing functionality with new packages. Unfortunately, I think this has created a cultural problem in which new packages are created that only slightly modify existing functionality.
  - longemen3000 6y ago
    One of the discussions on Juliacon was exactly this! How to index various things. This was one of the documents produced during the discussion : https://docs.google.com/document/d/1imBX3k0EEejauWVyXONZDRj8LTr0PeLOJNGEgo6ow1g/edit?usp=drivesdk https://docs.google.com/document/d/1imBX3k0EEejauWVyXONZDRj8...