5 ms·

But the runtime performance is not that great either. Dot product: NumPy 0.03528 vs 0.03 Julia (x1/1.1) Element-wise sum: NumPy 0.00379 vs 0.0063 J

But the runtime performance is not that great either. Dot product: NumPy 0.03528 vs 0.03 Julia (x1/1.1) Element-wise sum: NumPy 0.00379 vs 0.0063 Julia (x1.6) Element-wise mul: NumPy 0.00419 vs 0.00617 Julia (x1.5) L2 norm: NumPy 0.02391 vs 0.097 Julia (x4.1) Matrix product: NumPy 0.00186 vs 0.01988 Julia (x10.7) Matrix sort: NumPy 0.01033 vs 0.0161 Julia (x1.6)

tgv 6y ago
Numpy is written in C. What are Julia’s functions written in?
- tastyminerals2 6y ago
  Julia was written in C++ in early stages if I remember correctly then Julia ;) The point here is that NumPy is heavily fine-tuned and will beat almost any other language implementation out there. The community is bigger and therefore there are much more free hands.
  - pdeffebach 6y ago
    In Julia's case, there are maybe more "free hands" who can contribute to performance critical code. To contribute to `NumPy` you have to know how to program in C.
  - johnmyleswhite 6y ago
    One of the important facts about OSS development is that the number of users doesn't scale linearly with the number of "free hands. There are less than 2 FTE's working on Python at any given time these days.
kristofferc 6y ago
Aren't many of those benchmarks mostly comparing MKL vs OpenBLAS (the default BLAS library used)? Also, it's unlikely that Julia will handily beat the handwritten, carefully optimized C-code in numpy. At that level, it isn't very much about the language but more about the programmers implementing the algorithms. A major selling point of Julia is that you will have fast code even in cases where the code doesn't neatly factor into calling static kernels written in e.g. C.
- tastyminerals2 6y ago
  But this is also true for any other statically typed language. Java/Scala/Kotlin/Go/Rust you name it. Pure Java has a good chance of outperforming a mix of Python and NumPy. So what's Julia's selling point? Being dynamic and fast? But it comes with some really nasty side effects. Up to the point that it is sometimes doesn't even make sense and Go compile & execute combo doesn't feel much slower. At this stage I can only see that it is the ecosystem which is hard to evaluate without investing into learning the language. However, I find it hard to believe that it is richer than Java or Python.
  - eigenspace 6y ago
    > But this is also true for any other statically typed language. Java/Scala/Kotlin/Go/Rust you name it. Notice how none of those languages have a vibrant numerical computing ecosystem anywhere near Julia's despite being just as old or older than julia and being big, well known languages with large communities? It's almost like scientific programmers don't want to do science in those languages. I don't think it's an accident that Julia's package developers have been remarkably productive compared to Python. The best differnetial equations library available on Python right now is a wrapper for Julia's DifferentialEquations.jl. This is not because Python users don't care about differential equations or because there's not enough money or manpower. Chris Rackauckas did more on his own in julia for differential equation solving in his first couple years than entire industries did in Python over decades. Julia is a productive and performant language. When you hold up Java and Python, you're asking me to choose between productivity and performance (at best, obviously Python has serious productivity problems and Java has serious performance problems despite those being their touted strengths).
    - tastyminerals2 6y ago
      Come on. It's not like Julia has a PyTorch / TF equivalent (Flux.jl doesn't even come close, sorry). Until then, you can't seriously say that Julia's ecosystem is rich. Once we start seeing DL/ML papers coming out with POC written in Julia we can call it a day.
eigenspace 6y ago
Now try doing these tests on arrays of user defined types. Or use one of those fast functions as the kernel of an ODE solver and compare to julia. Anyone can game microbenchmarks on arrays of Float64s, doing so shouldn't impress anyone. If Numpy can do it, you know it's not impressive. The point of Julia is that Float64 and Int aren't special. They're on the same footing as whatever types you cook up, and our performant multiple dispatch system lets you build a vibrant interacting package ecosystem filled with user defined types doing targeted overloads of other functions. It's the multiple dispatch and composability of julia code that matters. The important part of the performance of Julia is that we can get the ecosystem benefits of writing pure julia code without paying with runtime performance penalties.
- mfsch 6y ago
  Even for floats & ints, try doing some array operations without allocating memory for intermediate results. In Julia, I can write whole PDE simulations that never allocate intermediate values inside the time integration loop. I haven’t done serious work with NumPy in a while, but I’m pretty sure that’s impossible for anything non-trivial, even a simple FD stencil such as `du[2:end-1] = D * (u[1:end-2] - 2*u[2:end-1] + u[3:end]) / dx^2`.
  - eigenspace 6y ago
    Absolutely. Numpy's performance model is so brittle that one needs to be very selective when they cherry-pick their microbenchmarks. My rule of thumb is that if your cherrypicked microbenchmark involves one specialized Numpy function call, it'll be fast, but if there is two separate specialized Numpy function calls it'll be slow.
pkofod 6y ago
This makes no sense without showing what you're running ? Are you also using MKL in Julia? https://github.com/JuliaComputing/MKL.jl https://github.com/JuliaComputing/MKL.jl
celrod 6y ago
I tried element-wise sum, and got julia> using LoopVectorization, BenchmarkTools julia> A = rand(10_000, 10_000); B = rand(10_000, 10_000); C = similar(A); julia> @benchmark vmapntt!(+, $C, $A, $B) BenchmarkTools.Trial: memory estimate: 13.19 KiB allocs estimate: 91 -------------- minimum time: 29.103 ms (0.00% GC) median time: 29.202 ms (0.00% GC) mean time: 29.329 ms (0.00% GC) maximum time: 30.658 ms (0.00% GC) -------------- samples: 171 evals/sample: 1 While with NumPy >>> import timeit >>> u = timeit.Timer("A + B", setup='import numpy as np; A = np.random.rand(10_000, 10_000); B = np.random.rand(10_000,10_000)') >>> u.repeat(10, 1) [0.17918606100283796, 0.17888473700440954, 0.17893354399711825, 0.1790916720055975, 0.17922663199715316, 0.17935074500564951, 0.17939399600436445, 0.17940548500337172, 0.17933111900492804, 0.17920518800383434] A 6x advantage for Julia. If I don't use multithreading, Julia slows down to 122.5ms, or about 1.5x faster. Elementwise multiplication is the same fast. If I get a little more creative and do exp(A) + log(B) instead, multithreaded Julia still takes 30ms, while single threaded slows down to 169ms. Meanwhile, NumPy slowed down to 847ms, making it 30x slower than multithreaded Julia (on my computer) and 5x slower than single threaded. Shall I keep making programs more complicated?
- tastyminerals2 6y ago
  Never use big arrays in benchmarks otherwise depending on your CPU cache you will be testing allocation as well. If you need the code I tested Julia on here you go: https://github.com/tastyminerals/mir_benchmarks/blob/master/other_benchmarks/julia_bench.jl https://github.com/tastyminerals/mir_benchmarks/blob/master/...
  - celrod 6y ago
    Thanks for sharing the benchmarks. I made the arrays huge to get roughly the same ball park of times you reported. I also preallocated the memory in that benchmark to avoid unnecessary allocations (the observed allocations are because multithreading in Julia allocates memory). Is this something as easy to do in Python? I made a few tweaks to your benchmark code. I get with Julia: Element-wise sum of two 100x100 matrices (int), (1000 loops) 1.698 ms (2 allocations: 78.20 KiB) Element-wise multiplication of two 100x100 matrices (float64), (1000 loops) 1.786 ms (2 allocations: 78.20 KiB) Dot (scalar) product of two 300000 arrays (float64), (1000 loops) 4.801 ms (0 allocations: 0 bytes) Matrix product of 500x600 and 600x500 matrices (float64) 229.475 μs (0 allocations: 0 bytes) L2 norm of 500x600 matrix (float64), (1000 loops) 6.566 ms (0 allocations: 0 bytes) Sort of 500x600 matrix (float64) 523.243 μs (595 allocations: 2.32 MiB) Python: | Element-wise sum of two 100x100 matrices (int), (1000 loops) | 0.0042680892984208185 | | Element-wise multiplication of two 100x100 matrices (float64), (1000 loops) | 0.0027007726486772297 | | Dot (scalar) product of two 300000 arrays (float64), (1000 loops) | 0.0088505959516624 | | Matrix product of 500x600 and 600x500 matrices (float64) | 0.00107754109922098 | | L2 norm of 500x600 matrix (float64), (1000 loops) | 0.010691180499998154 | | Sort of 500x600 matrix (float64) | 0.008244920199649642 | That is, Julia-speedup factors of: 1. Small int matrix element-wise sum: 2.5x 2. Small float matrix element-wise product: 1.5x 3. Dot product (scalar): 1.84x 4. Matrix multiplication: 4.7x 5. L2 norm: 1.6x 6. Sorting: 15.8x The changes were: 1. Memory allocation was slow, but Julia makes in-place operations easy, so I made 1, 2, and 4 in-place. Although I still allocate the result within the test function for `1.` and `2.`. 2. MKL is much faster than OpenBLAS, so I made `BLAS.vendor()` return `:mkl` 3. I sorted in a threaded loop and transposed the destination array. Transposing the destination array was because Julia is column major, so sorting rows is a benchmark that will naturally favor the row-major language. But I left all the inputs the same. My primary point with Julia here is that it's easy to optimize it and get more performance out of it if or whenever you need it.

tgv 6y ago

Numpy is written in C. What are Julia’s functions written in?

tastyminerals2 6y ago
Julia was written in C++ in early stages if I remember correctly then Julia ;) The point here is that NumPy is heavily fine-tuned and will beat almost any other language implementation out there. The community is bigger and therefore there are much more free hands.
- pdeffebach 6y ago
  In Julia's case, there are maybe more "free hands" who can contribute to performance critical code. To contribute to `NumPy` you have to know how to program in C.
- johnmyleswhite 6y ago
  One of the important facts about OSS development is that the number of users doesn't scale linearly with the number of "free hands. There are less than 2 FTE's working on Python at any given time these days.

kristofferc 6y ago

Aren't many of those benchmarks mostly comparing MKL vs OpenBLAS (the default BLAS library used)? Also, it's unlikely that Julia will handily beat the handwritten, carefully optimized C-code in numpy. At that level, it isn't very much about the language but more about the programmers implementing the algorithms. A major selling point of Julia is that you will have fast code even in cases where the code doesn't neatly factor into calling static kernels written in e.g. C.

tastyminerals2 6y ago
But this is also true for any other statically typed language. Java/Scala/Kotlin/Go/Rust you name it. Pure Java has a good chance of outperforming a mix of Python and NumPy. So what's Julia's selling point? Being dynamic and fast? But it comes with some really nasty side effects. Up to the point that it is sometimes doesn't even make sense and Go compile & execute combo doesn't feel much slower. At this stage I can only see that it is the ecosystem which is hard to evaluate without investing into learning the language. However, I find it hard to believe that it is richer than Java or Python.
- eigenspace 6y ago
  > But this is also true for any other statically typed language. Java/Scala/Kotlin/Go/Rust you name it. Notice how none of those languages have a vibrant numerical computing ecosystem anywhere near Julia's despite being just as old or older than julia and being big, well known languages with large communities? It's almost like scientific programmers don't want to do science in those languages. I don't think it's an accident that Julia's package developers have been remarkably productive compared to Python. The best differnetial equations library available on Python right now is a wrapper for Julia's DifferentialEquations.jl. This is not because Python users don't care about differential equations or because there's not enough money or manpower. Chris Rackauckas did more on his own in julia for differential equation solving in his first couple years than entire industries did in Python over decades. Julia is a productive and performant language. When you hold up Java and Python, you're asking me to choose between productivity and performance (at best, obviously Python has serious productivity problems and Java has serious performance problems despite those being their touted strengths).
  - tastyminerals2 6y ago
    Come on. It's not like Julia has a PyTorch / TF equivalent (Flux.jl doesn't even come close, sorry). Until then, you can't seriously say that Julia's ecosystem is rich. Once we start seeing DL/ML papers coming out with POC written in Julia we can call it a day.

eigenspace 6y ago

Now try doing these tests on arrays of user defined types. Or use one of those fast functions as the kernel of an ODE solver and compare to julia. Anyone can game microbenchmarks on arrays of Float64s, doing so shouldn't impress anyone. If Numpy can do it, you know it's not impressive. The point of Julia is that Float64 and Int aren't special. They're on the same footing as whatever types you cook up, and our performant multiple dispatch system lets you build a vibrant interacting package ecosystem filled with user defined types doing targeted overloads of other functions. It's the multiple dispatch and composability of julia code that matters. The important part of the performance of Julia is that we can get the ecosystem benefits of writing pure julia code without paying with runtime performance penalties.

mfsch 6y ago
Even for floats & ints, try doing some array operations without allocating memory for intermediate results. In Julia, I can write whole PDE simulations that never allocate intermediate values inside the time integration loop. I haven’t done serious work with NumPy in a while, but I’m pretty sure that’s impossible for anything non-trivial, even a simple FD stencil such as `du[2:end-1] = D * (u[1:end-2] - 2*u[2:end-1] + u[3:end]) / dx^2`.
- eigenspace 6y ago
  Absolutely. Numpy's performance model is so brittle that one needs to be very selective when they cherry-pick their microbenchmarks. My rule of thumb is that if your cherrypicked microbenchmark involves one specialized Numpy function call, it'll be fast, but if there is two separate specialized Numpy function calls it'll be slow.

pkofod 6y ago

This makes no sense without showing what you're running ? Are you also using MKL in Julia? https://github.com/JuliaComputing/MKL.jl https://github.com/JuliaComputing/MKL.jl

celrod 6y ago

I tried element-wise sum, and got julia> using LoopVectorization, BenchmarkTools julia> A = rand(10_000, 10_000); B = rand(10_000, 10_000); C = similar(A); julia> @benchmark vmapntt!(+, $C, $A, $B) BenchmarkTools.Trial: memory estimate: 13.19 KiB allocs estimate: 91 -------------- minimum time: 29.103 ms (0.00% GC) median time: 29.202 ms (0.00% GC) mean time: 29.329 ms (0.00% GC) maximum time: 30.658 ms (0.00% GC) -------------- samples: 171 evals/sample: 1 While with NumPy >>> import timeit >>> u = timeit.Timer("A + B", setup='import numpy as np; A = np.random.rand(10_000, 10_000); B = np.random.rand(10_000,10_000)') >>> u.repeat(10, 1) [0.17918606100283796, 0.17888473700440954, 0.17893354399711825, 0.1790916720055975, 0.17922663199715316, 0.17935074500564951, 0.17939399600436445, 0.17940548500337172, 0.17933111900492804, 0.17920518800383434] A 6x advantage for Julia. If I don't use multithreading, Julia slows down to 122.5ms, or about 1.5x faster. Elementwise multiplication is the same fast. If I get a little more creative and do exp(A) + log(B) instead, multithreaded Julia still takes 30ms, while single threaded slows down to 169ms. Meanwhile, NumPy slowed down to 847ms, making it 30x slower than multithreaded Julia (on my computer) and 5x slower than single threaded. Shall I keep making programs more complicated?

tastyminerals2 6y ago
Never use big arrays in benchmarks otherwise depending on your CPU cache you will be testing allocation as well. If you need the code I tested Julia on here you go: https://github.com/tastyminerals/mir_benchmarks/blob/master/other_benchmarks/julia_bench.jl https://github.com/tastyminerals/mir_benchmarks/blob/master/...
- celrod 6y ago
  Thanks for sharing the benchmarks. I made the arrays huge to get roughly the same ball park of times you reported. I also preallocated the memory in that benchmark to avoid unnecessary allocations (the observed allocations are because multithreading in Julia allocates memory). Is this something as easy to do in Python? I made a few tweaks to your benchmark code. I get with Julia: Element-wise sum of two 100x100 matrices (int), (1000 loops) 1.698 ms (2 allocations: 78.20 KiB) Element-wise multiplication of two 100x100 matrices (float64), (1000 loops) 1.786 ms (2 allocations: 78.20 KiB) Dot (scalar) product of two 300000 arrays (float64), (1000 loops) 4.801 ms (0 allocations: 0 bytes) Matrix product of 500x600 and 600x500 matrices (float64) 229.475 μs (0 allocations: 0 bytes) L2 norm of 500x600 matrix (float64), (1000 loops) 6.566 ms (0 allocations: 0 bytes) Sort of 500x600 matrix (float64) 523.243 μs (595 allocations: 2.32 MiB) Python: | Element-wise sum of two 100x100 matrices (int), (1000 loops) | 0.0042680892984208185 | | Element-wise multiplication of two 100x100 matrices (float64), (1000 loops) | 0.0027007726486772297 | | Dot (scalar) product of two 300000 arrays (float64), (1000 loops) | 0.0088505959516624 | | Matrix product of 500x600 and 600x500 matrices (float64) | 0.00107754109922098 | | L2 norm of 500x600 matrix (float64), (1000 loops) | 0.010691180499998154 | | Sort of 500x600 matrix (float64) | 0.008244920199649642 | That is, Julia-speedup factors of: 1. Small int matrix element-wise sum: 2.5x 2. Small float matrix element-wise product: 1.5x 3. Dot product (scalar): 1.84x 4. Matrix multiplication: 4.7x 5. L2 norm: 1.6x 6. Sorting: 15.8x The changes were: 1. Memory allocation was slow, but Julia makes in-place operations easy, so I made 1, 2, and 4 in-place. Although I still allocate the result within the test function for `1.` and `2.`. 2. MKL is much faster than OpenBLAS, so I made `BLAS.vendor()` return `:mkl` 3. I sorted in a threaded loop and transposed the destination array. Transposing the destination array was because Julia is column major, so sorting rows is a benchmark that will naturally favor the row-major language. But I left all the inputs the same. My primary point with Julia here is that it's easy to optimize it and get more performance out of it if or whenever you need it.