4 ms·

I feel like you (and them) are still missing the point. 99% of users don't write GPU kernels. They use simple function calls and that's all they see. 99% of use

by p1esk 6y ago

I feel like you (and them) are still missing the point. 99% of users don't write GPU kernels. They use simple function calls and that's all they see. 99% of users don't care if they call CUDA code from Python, or Julia code from Julia, as long as both are equally fast and API is clean (e.g. Pytorch).

ed_elliott_asc 6y ago
I watched an interview with him, his end goal is no code machine learning so when he is saying python isn’t the future it is because he prefers a type system that can’t be achieved with python. I agree that users don’t care, I use Spark a lot and don’t care if I write scala, python or even c# now. Have a listen: https://www.wandb.com/podcast/jeremy-howard https://www.wandb.com/podcast/jeremy-howard (I think I have paraphrased the podcast correctly :) )
- p1esk 6y ago
  end goal is no code machine learning Now that I agree on. I suspect we will see something like GPT-5 reliably converting plain English into code sooner than Julia displacing Python as king of ML.
layoric 6y ago
I don’t think they are missing the point. These abstractions are far from perfect and trying to hunt down why something is not performing and tracing it back to how syntax is being interpreted as data is stored (GPU memory vs system memory vs sent over the pcie bus) or passed between other glue code is a source of frustration I’m sure many people have experienced. It will take a lot for another language eco system to catch-up to something as broad and deep as python but these issues combined with the horrible experience that is dependency management with pure python vs compiled etc as well as the need for real-time/close to real-time systems, it seems very plausible that something like Julia or other llvm language will become the default over python for _these specific_ use cases, not in general though. Tooling on top of Julia can likely be far better and more easily made than living in python which is another aspect.
- p1esk 6y ago
  trying to hunt down why something is not performing Again, 99% of the users won't be doing that no matter how good the tracing abilities are. They will post a question on SO/GitHub. Regardless of the language, modern DL frameworks are just too complex for regular users.
dmarchand90 6y ago
Perhaps the fact that 99% of users never write kernels is partial due to the fact that you need to switch languages to do so. Maybe a lot more users might want to write and modify GPU kernels if they had that as an easier option
- p1esk 6y ago
  Writing a fast CUDA kernel requires quite a bit of low level GPU knowledge. You typically do that when you really need it and it's not already available - like when you invented a new sparse GEMM algorithm, or want to speed up convolutions with binary weights/inputs. If you do need that, congrats, you're the 1%, and you're likely to not be afraid of mixing languages and getting your hands dirty with C++.
  - pjmlp 6y ago
    One of the reasons CUDA wipedout OpenCL is due to its polyglot support, no need to get hands dirty with C++. Fortran, Java, .NET. Julia, D, Rust and anything else able to either generate PTX or integrate with CUDA LLVM backend can be used.
  - amkkma 6y ago
    >Writing a fast CUDA kernel requires quite a bit of low level GPU knowledge False. Not in Julia: https://juliagpu.gitlab.io/KernelAbstractions.jl/ https://juliagpu.gitlab.io/KernelAbstractions.jl/ https://github.com/mcabbott/Tullio.jl https://github.com/mcabbott/Tullio.jl
    - ChrisRackauckas 6y ago
      To share the details, here's a post demonstrating Tullio.jl, a Julia-based higher order tensor library, is faster than OpenBLAS for gemm kernels: https://discourse.julialang.org/t/realistically-how-close-is-gaius-jl-to-becoming-a-full-replacement-for-blas-in-julia/42780/5 https://discourse.julialang.org/t/realistically-how-close-is... And PaddedMatrices.jl is another example: https://discourse.julialang.org/t/ann-paddedmatrices-jl-julia-blas-and-partially-sized-arrays/38215 https://discourse.julialang.org/t/ann-paddedmatrices-jl-juli... So there isn't just one example but multiples at this point. RecursiveFactorization's `lu` tends to perform better than MKL for small matrices and this is what we use in the differential equation solvers when there are <100 ODEs. That's all on CPUs though. KernelAbstractions.jl is the GPU version and that's still being constructed but moving strong.
dragandj 6y ago
This is because 99% of users know only Python, and C++ is too much for them. If the advanced stuff was available in the language they use, larger portion of users would write more advanced stuff. I prefer Clojure for this. Even simpler than Python, fast, and enables me to match or even surpass the speed expected from mainstream tools. See this for example: https://dragan.rocks/articles/20/Deep-Diamond-Deep-Learning-in-Clojure-is-Fast-and-Simpler-than-Keras https://dragan.rocks/articles/20/Deep-Diamond-Deep-Learning-...
- p1esk 6y ago
  Did you write gpu kernels in clojure? Or did you make function calls to cudnn like pytorch et al does it?
  - dragandj 6y ago
    Both. See https://github.com/uncomplicate/bayadera https://github.com/uncomplicate/bayadera too. Whatever the language, you have to make function calls to cuDNN, because cuDNN is optimized by Nvidia, which put serious resources into it. Even if you wrote everything in C++/CUDA, you probably can't match it for the standard stuff. The custom things that I write, I prefer to write in CUDA kernels + Clojure management code. Clojure kernels are technically possible in the same way as Julia or whatever kernels are possible, but the thing is that you have to match low-level hardware features anyway, so CUDA C/C++ just makes more sense for kernels, which are tiny part of code anyway.
    - p1esk 6y ago
      So if you have to use cuda c/c++, then why not do it from python?
      - dragandj 6y ago
        No. You don't use cuda/c++ as it is used from Python. You use cuda/c only in kernels. That is compiled dynamically, in process, from Clojure funcion calls, and 95% of related code is the management code, which is written in Clojure. There is no separate C++ build. You never touch any C++ tools such as CMake or anything. You work dynamically from the Clojure REPL.
ummonk 6y ago
And notably, CUDA.jl or whatever Julia package you use for gpu acceleration still has to incorporate CUDA, the same way Pytorch does. So to the end user who doesn't want to write CUDA, I don't see why using a Julia package for gpu acceleration is any better than using PyTorch in Python.
- amkkma 6y ago
  Wrong. Julia can call out to Cuda kernels, but it also does codegen for custom ops and types. Technically with enough engineering time, someone could rewrite all those C cuda kernels in julia, and julia would just generate low level machine GPU code on the fly. Pytorch on the other hand HAS to glue together c and c++ cuda code.
  - ummonk 6y ago
    You mean like PyCuda?
    - dklend122 6y ago
      Nope. PyCuda is writing inline C cuda and hardly integrated with python. Julia has four layers of GPU abstractions, each being successively easier to use ( bit a bit less control. ), all in pure Julia and reusing the compiler and normal Julia abstractions and integrated with AD etc CUDA.JL, GPUArrays.jl, kernelabstracions.jl, tullio.jl. At the highest levels you just write normal loops or index notation and the code and backward pass generated. Also generic to backend-simd and multithreading cpu and soon AMD rock, intel accelerator and distributed This all composed with deep learning libs etc

by p1esk 6y ago

ed_elliott_asc 6y ago

I watched an interview with him, his end goal is no code machine learning so when he is saying python isn’t the future it is because he prefers a type system that can’t be achieved with python. I agree that users don’t care, I use Spark a lot and don’t care if I write scala, python or even c# now. Have a listen: https://www.wandb.com/podcast/jeremy-howard https://www.wandb.com/podcast/jeremy-howard (I think I have paraphrased the podcast correctly :) )

p1esk 6y ago
end goal is no code machine learning Now that I agree on. I suspect we will see something like GPT-5 reliably converting plain English into code sooner than Julia displacing Python as king of ML.

layoric 6y ago

I don’t think they are missing the point. These abstractions are far from perfect and trying to hunt down why something is not performing and tracing it back to how syntax is being interpreted as data is stored (GPU memory vs system memory vs sent over the pcie bus) or passed between other glue code is a source of frustration I’m sure many people have experienced. It will take a lot for another language eco system to catch-up to something as broad and deep as python but these issues combined with the horrible experience that is dependency management with pure python vs compiled etc as well as the need for real-time/close to real-time systems, it seems very plausible that something like Julia or other llvm language will become the default over python for _these specific_ use cases, not in general though. Tooling on top of Julia can likely be far better and more easily made than living in python which is another aspect.

p1esk 6y ago
trying to hunt down why something is not performing Again, 99% of the users won't be doing that no matter how good the tracing abilities are. They will post a question on SO/GitHub. Regardless of the language, modern DL frameworks are just too complex for regular users.

dmarchand90 6y ago

Perhaps the fact that 99% of users never write kernels is partial due to the fact that you need to switch languages to do so. Maybe a lot more users might want to write and modify GPU kernels if they had that as an easier option

p1esk 6y ago
Writing a fast CUDA kernel requires quite a bit of low level GPU knowledge. You typically do that when you really need it and it's not already available - like when you invented a new sparse GEMM algorithm, or want to speed up convolutions with binary weights/inputs. If you do need that, congrats, you're the 1%, and you're likely to not be afraid of mixing languages and getting your hands dirty with C++.
- pjmlp 6y ago
  One of the reasons CUDA wipedout OpenCL is due to its polyglot support, no need to get hands dirty with C++. Fortran, Java, .NET. Julia, D, Rust and anything else able to either generate PTX or integrate with CUDA LLVM backend can be used.
- amkkma 6y ago
  >Writing a fast CUDA kernel requires quite a bit of low level GPU knowledge False. Not in Julia: https://juliagpu.gitlab.io/KernelAbstractions.jl/ https://juliagpu.gitlab.io/KernelAbstractions.jl/ https://github.com/mcabbott/Tullio.jl https://github.com/mcabbott/Tullio.jl
  - ChrisRackauckas 6y ago
    To share the details, here's a post demonstrating Tullio.jl, a Julia-based higher order tensor library, is faster than OpenBLAS for gemm kernels: https://discourse.julialang.org/t/realistically-how-close-is-gaius-jl-to-becoming-a-full-replacement-for-blas-in-julia/42780/5 https://discourse.julialang.org/t/realistically-how-close-is... And PaddedMatrices.jl is another example: https://discourse.julialang.org/t/ann-paddedmatrices-jl-julia-blas-and-partially-sized-arrays/38215 https://discourse.julialang.org/t/ann-paddedmatrices-jl-juli... So there isn't just one example but multiples at this point. RecursiveFactorization's `lu` tends to perform better than MKL for small matrices and this is what we use in the differential equation solvers when there are <100 ODEs. That's all on CPUs though. KernelAbstractions.jl is the GPU version and that's still being constructed but moving strong.

dragandj 6y ago

This is because 99% of users know only Python, and C++ is too much for them. If the advanced stuff was available in the language they use, larger portion of users would write more advanced stuff. I prefer Clojure for this. Even simpler than Python, fast, and enables me to match or even surpass the speed expected from mainstream tools. See this for example: https://dragan.rocks/articles/20/Deep-Diamond-Deep-Learning-in-Clojure-is-Fast-and-Simpler-than-Keras https://dragan.rocks/articles/20/Deep-Diamond-Deep-Learning-...

p1esk 6y ago
Did you write gpu kernels in clojure? Or did you make function calls to cudnn like pytorch et al does it?
- dragandj 6y ago
  Both. See https://github.com/uncomplicate/bayadera https://github.com/uncomplicate/bayadera too. Whatever the language, you have to make function calls to cuDNN, because cuDNN is optimized by Nvidia, which put serious resources into it. Even if you wrote everything in C++/CUDA, you probably can't match it for the standard stuff. The custom things that I write, I prefer to write in CUDA kernels + Clojure management code. Clojure kernels are technically possible in the same way as Julia or whatever kernels are possible, but the thing is that you have to match low-level hardware features anyway, so CUDA C/C++ just makes more sense for kernels, which are tiny part of code anyway.
  - p1esk 6y ago
    So if you have to use cuda c/c++, then why not do it from python?
    - dragandj 6y ago
      No. You don't use cuda/c++ as it is used from Python. You use cuda/c only in kernels. That is compiled dynamically, in process, from Clojure funcion calls, and 95% of related code is the management code, which is written in Clojure. There is no separate C++ build. You never touch any C++ tools such as CMake or anything. You work dynamically from the Clojure REPL.

ummonk 6y ago

And notably, CUDA.jl or whatever Julia package you use for gpu acceleration still has to incorporate CUDA, the same way Pytorch does. So to the end user who doesn't want to write CUDA, I don't see why using a Julia package for gpu acceleration is any better than using PyTorch in Python.

amkkma 6y ago
Wrong. Julia can call out to Cuda kernels, but it also does codegen for custom ops and types. Technically with enough engineering time, someone could rewrite all those C cuda kernels in julia, and julia would just generate low level machine GPU code on the fly. Pytorch on the other hand HAS to glue together c and c++ cuda code.
- ummonk 6y ago
  You mean like PyCuda?
  - dklend122 6y ago
    Nope. PyCuda is writing inline C cuda and hardly integrated with python. Julia has four layers of GPU abstractions, each being successively easier to use ( bit a bit less control. ), all in pure Julia and reusing the compiler and normal Julia abstractions and integrated with AD etc CUDA.JL, GPUArrays.jl, kernelabstracions.jl, tullio.jl. At the highest levels you just write normal loops or index notation and the code and backward pass generated. Also generic to backend-simd and multithreading cpu and soon AMD rock, intel accelerator and distributed This all composed with deep learning libs etc