4 ms·

That said, the Julia community has done some pretty amazing work in that subfield, last I checked they were getting performance in pure-Julia very close to the

That said, the Julia community has done some pretty amazing work in that subfield, last I checked they were getting performance in pure-Julia very close to the of BLAS/LAPACK, which is pretty incredible.

eigenspace 6y ago
https://discourse.julialang.org/t/realistically-how-close-is-gaius-jl-to-becoming-a-full-replacement-for-blas-in-julia/42780/5 https://discourse.julialang.org/t/realistically-how-close-is...
xiphias2 6y ago
Julia is improving, but I wouldn't compare it to BLAS/LAPACK, cuBLAS is a much better comparision. PyTorch uses tensor cores, supports mixed precision multiplication and bfloat16 representation. Julia's implementation in CUDA.jl doesn't even use shared memory, which is important, as using that 10MB cache can reduce the needed main memory bandwidth significantly. Also the new 30xx NVIDIA series supports sparse matrix multiplication, which is great if you have ReLU operations in your data processing for example. I'm not sure if PyTorch has support for that, but I'm sure it will be added, just like how mixed precision multiplication was added this year. We're talking about an order of magnitude difference if you add all these small things together that PyTorch / cuBLAS supports. Of course Julia can call into these libraries, but then all the nice optimizations that make Julia Julia go away. I had to think really hard of these things actually, because I love the Julia language, but the more I see ML pipelines beating the previous state-of-the-art models by 5x, the more I understand why an interpreted language with a great linear algebra library won the data processing race (I wish it would have been Ruby though).
- dklend122 6y ago
  No, the parent comment is correct. Tulllio.jl has been getting up to openBLAS levels of performance Julia also is on the cusp of support for tensor cores, mixed precision multiplication and bfloat16 codegen. New PR just merged last week
  - xiphias2 6y ago
    https://discourse.julialang.org/t/blas-vs-cublas-benchmark/46205/3 https://discourse.julialang.org/t/blas-vs-cublas-benchmark/4... Here's an OpenBLAS vs CuBLAS performance comparision. Basically CPU is at this point outdated technology for matrix multiplication. Your second comment is more interesting, I'd be happy if Julia gets to the point when it can beat PyTorch or even JAX on training Lambda networks or Performers.
    - dklend122 6y ago
      I understand CPU is slower, but it's not going anywhere, and matmul is just a benchmark. It's extremely impressive that a high level (using index notation for multiple backends) Julia library can compete with hand tuned kernels. Also bodes well for Julia compiler tech generally that can easily be extended Julia is going to get to the point of beating pytorch, just taking a but more time due to smaller team and approaching it from a more general position..so when it does it will be more flexible and ergonomic and easily extensible to new techniques, in pure Julia. 1.6 will be a big step as much of the compiler hacks underlying the current ad/gpu codegen (which just fell out accidentally of lispy design) will be replaced with proper tooling for composable compiler passes on typed IR. This will be a phase change imo. There's already a new faster AD that's almost ready for debut based on 1.6 tech
      - xiphias2 6y ago
        That sounds amazing, thanks for the update! I stopped using Julia because it was very frustrating that I payed thousands of dollars for a laptop with GeForce RTX 2070 card and the I couldn't make use of it, it felt like I wasted a lot of money...that was my main decision for moving to PyTorch. I know that Julia as a language should be able to beat it easily, but it felt to me that the community was prioritizing CPU over GPU for a long time. I'm happy that it's changing now.
    - eigenspace 6y ago
      The GPU stuff is basically just a teaser into the future, it's still much more immature than that CPU stuff for Tullio. > Basically CPU is at this point outdated technology for matrix multiplication. This is an incredibly naive statement. There are many many circumstances where you need fast matmul on the CPU even if you have a GPU available because it takes too long to send the memory to the GPU and fetch it once the kernel runs. If you have many chained matmuls, then yes the GPU is your friend. That said, regardless of how useful the CPU is in practice for matrix multiplication, BLAS kernels are some of the most overengineered, most optimized pieces of code in wide use. Tullio is incredibly flexibly and general. The fact that matrix multiplication falls out as a special case in Tullio and can outperform OpenBLAS and match MKL is absolutely stunning.
physicsguy 6y ago
Which implementation of BLAS/LAPACK? Because if it's the ref. implementation then it's not that impressive, the performance of that is very poor.
- eigenspace 6y ago
  OpenBLAS and MKL.

eigenspace 6y ago

https://discourse.julialang.org/t/realistically-how-close-is-gaius-jl-to-becoming-a-full-replacement-for-blas-in-julia/42780/5 https://discourse.julialang.org/t/realistically-how-close-is...

xiphias2 6y ago

Julia is improving, but I wouldn't compare it to BLAS/LAPACK, cuBLAS is a much better comparision. PyTorch uses tensor cores, supports mixed precision multiplication and bfloat16 representation. Julia's implementation in CUDA.jl doesn't even use shared memory, which is important, as using that 10MB cache can reduce the needed main memory bandwidth significantly. Also the new 30xx NVIDIA series supports sparse matrix multiplication, which is great if you have ReLU operations in your data processing for example. I'm not sure if PyTorch has support for that, but I'm sure it will be added, just like how mixed precision multiplication was added this year. We're talking about an order of magnitude difference if you add all these small things together that PyTorch / cuBLAS supports. Of course Julia can call into these libraries, but then all the nice optimizations that make Julia Julia go away. I had to think really hard of these things actually, because I love the Julia language, but the more I see ML pipelines beating the previous state-of-the-art models by 5x, the more I understand why an interpreted language with a great linear algebra library won the data processing race (I wish it would have been Ruby though).

dklend122 6y ago
No, the parent comment is correct. Tulllio.jl has been getting up to openBLAS levels of performance Julia also is on the cusp of support for tensor cores, mixed precision multiplication and bfloat16 codegen. New PR just merged last week
- xiphias2 6y ago
  https://discourse.julialang.org/t/blas-vs-cublas-benchmark/46205/3 https://discourse.julialang.org/t/blas-vs-cublas-benchmark/4... Here's an OpenBLAS vs CuBLAS performance comparision. Basically CPU is at this point outdated technology for matrix multiplication. Your second comment is more interesting, I'd be happy if Julia gets to the point when it can beat PyTorch or even JAX on training Lambda networks or Performers.
  - dklend122 6y ago
    I understand CPU is slower, but it's not going anywhere, and matmul is just a benchmark. It's extremely impressive that a high level (using index notation for multiple backends) Julia library can compete with hand tuned kernels. Also bodes well for Julia compiler tech generally that can easily be extended Julia is going to get to the point of beating pytorch, just taking a but more time due to smaller team and approaching it from a more general position..so when it does it will be more flexible and ergonomic and easily extensible to new techniques, in pure Julia. 1.6 will be a big step as much of the compiler hacks underlying the current ad/gpu codegen (which just fell out accidentally of lispy design) will be replaced with proper tooling for composable compiler passes on typed IR. This will be a phase change imo. There's already a new faster AD that's almost ready for debut based on 1.6 tech
    - xiphias2 6y ago
      That sounds amazing, thanks for the update! I stopped using Julia because it was very frustrating that I payed thousands of dollars for a laptop with GeForce RTX 2070 card and the I couldn't make use of it, it felt like I wasted a lot of money...that was my main decision for moving to PyTorch. I know that Julia as a language should be able to beat it easily, but it felt to me that the community was prioritizing CPU over GPU for a long time. I'm happy that it's changing now.
  - eigenspace 6y ago
    The GPU stuff is basically just a teaser into the future, it's still much more immature than that CPU stuff for Tullio. > Basically CPU is at this point outdated technology for matrix multiplication. This is an incredibly naive statement. There are many many circumstances where you need fast matmul on the CPU even if you have a GPU available because it takes too long to send the memory to the GPU and fetch it once the kernel runs. If you have many chained matmuls, then yes the GPU is your friend. That said, regardless of how useful the CPU is in practice for matrix multiplication, BLAS kernels are some of the most overengineered, most optimized pieces of code in wide use. Tullio is incredibly flexibly and general. The fact that matrix multiplication falls out as a special case in Tullio and can outperform OpenBLAS and match MKL is absolutely stunning.

physicsguy 6y ago

Which implementation of BLAS/LAPACK? Because if it's the ref. implementation then it's not that impressive, the performance of that is very poor.

eigenspace 6y ago
OpenBLAS and MKL.