9 ms·

Comparing Performance of Julia on CPUs vs. GPUs and Julia-MPI vs. Fortran-MPI

95 points by npalli 3y ago

dataangel 3y ago
TL;DR - Julia nice for prototyping - but zero speed up from using GPU - sometimes faster sometimes slower on CPU - no plans to use at the moment
- oivey 3y ago
  This is a little reductive. The speed comparison was between the highly parallel and optimized CPU version vs GPU. The GPU’s advertised FLOPS was only 4x the CPU setup’s. That the performance is so similar isn’t all that surprising and could come down to low level optimization details. There are likely some areas in Julia’s GPU stack that could be made better, but it’s certainly already pretty performant. It’s also comparing against Fortran. Their Python comparison is 100x+ slower. That a high level language even comes close is impressive.
  - jppittma 3y ago
    If only they didn’t chose 1 based array indexing, like matlab. Having worked with matlab and for that reason I can’t seem to take them seriously.
    - baggepinnen 3y ago
      But you can take Fortran seriously?
    - pjmlp 3y ago
      Plenty of serious languages have either 1 based indexing, or allow the flexibility to set lower boundary explicitly.
    - marmalade2413 3y ago
      The choice of 1 based array indexing is due to Julia having it's roots in mathematics - which is a common choice for mathematics based programming languages (e.g. Fortran, Julia, MATLAB/GNU Octave, Mathematica, R and Wolfram). The reason being is that matrix notation starts at 1 and not 0. For a comparison on array indexing in different programming languages see here: https://en.wikipedia.org/wiki/Comparison_of_programming_languages_(array)#Array_dimensions https://en.wikipedia.org/wiki/Comparison_of_programming_lang... From personal experience, I wouldn't let something as simple as starting index getting in the way of programming for a language as you'll soon start automatically switching to thinking in 1 based and 0 based indexing depending on the programming language.
    - amj7e 3y ago
      If you think 0 or 1 based indexing will hinder your ability to write a program, it's not the indexing, it's a skill issue.
  - pjmlp 3y ago
    Why should it be impressive? Compiled high level languages have always been pretty fast in general, for decades, before 20 years ago it started to be trendy to use interpreted scripting languages to do their work. Anyone used to Modula-2, Object Pascal, QuickBasic, Clipper, AMOS, Lisp, Scheme, Dylan, Smalltalk, SELF, Oberon, VB, Delphi, C++, SML, Miranda, Haskell, Prolog from 1980-1990's, isn't going to be surprised about speedups over Python.
    - oivey 3y ago
      100x relative to NumPy is still quite a lot, depending on how much of their code manages to stay in the C extension. If it was pure Python the delta would likely be much, much larger. Julia is definitely built on the shoulders of giants. I don’t have experience with most of those languages, especially in high performance Lisp or Haskell. Calling C++ high level is definitely not in the same vein as Python or Julia. My understanding is that Haskell can run into all sorts of performance gotchas because lazy evaluation can be unpredictable and recursion may not always easily unroll into optimizable loops. I think the combination of features, especially borrowed from Lisp and C++, really come together as something very productive for writing fast, very high level code.
      - pjmlp 3y ago
        C++ is definitely high level, in the same vein as Python and Julia, the only thing missing is reflection (if we ignore C++/CLI, C++ Builder and custom libraries), which with some luck might still make the cut to C++26. Just because some people insist in using it like C, doesn't make it less higher level in available features and tooling. Regardless of how easy it is to optimize Haskell code, even stupid Haskell code beats an interpreted language like Python. And C bindings are not Python, regardless of how the Python community likes to think otherwise. If anything, I look forward to Julia and others to put pressure in the community to take PyPy and other JIT attempts more seriously. Or maybe it doesn't matter and in a couple of years everyone has moved into mojo, if it is successful.
        oivey 3y ago
npalli 3y ago
Ok, so a not very representative tl;dr was provided so adding mine here. 1. Serial CPU performance of Julia-unoptimized was around 12 times that of Python/Numpy. 2. Serial CPU performance of Julia optimized gave another 10-12x improvement leading to total 140x improvement over Python/Numpy. (good pointers on speeding up the Python code beyond Numpy listed but not tried or benchmarked) 3. Even though the GPU was rated for a listed performance of 9.7TFlops (A100) the code performed around the same as the optimized Serial CPU which has a listed performance of 2.5TFlops over 64 cores. Surprising result, there are some pointers to why that might be. 4. When comparing Julia-MPI with Fortran-MPI, at lower processor counts both match for total time, at middle processor counts (16-128) Fortran takes more time than Julia, at higher processor counts Julia becomes slower than Fortran. However the scaling down of Julia is driven more by a less efficient implementation of the communication time with the higher processor setup. Computation time scales well for both Julia and Fortran but the communication time takes over in the Julia case. Some nice optimization tips are provided to speed up Julia code. No plans to use Julia yet but it is good for prototyping and Fortran is good enough for now.
- noobermin 3y ago
  Just to be clear, doing well compared to an establish fortran code is an achievement. An established code is THE reference point.
  - npalli 3y ago
    Yes, definitely very impressive results. Especially considering how much of an improvement over the Python/Numpy code they could achieve. If you are writing something from scratch, very good reason to work in Julia.
    - guenthert 3y ago
      Uhm, the poor performance of the Python/Numpy code indicates that there's something amiss; perhaps an apple/orange comparison. Don't get me wrong, Python is slow as molasses, but for this kind of applications, almost all time spent is (ought to be) spent in the Numpy code, which is mostly C/C++. Other systems might perform better, but I don't expect more than a factor of two (which itself would be huge in the HPC domain). Worth having a closer look methinks.
qrios 3y ago
For sure another case for the "you doing it wrong" team at Nvidia. But the decision to add Julia/CUDA.jl here was not the best idea on a comparison with really interesting insights about Fortran vs. Julia. The CUDA part of the code is nearly a one-to-one copy of the MPI code. Especially one sentence[1] (there are some similar, comparable blocks) explains why the code not performs very well. [1] https://github.com/robertstrauss/MPAS_Ocean_Julia/blob/main/mode_forward/calculate_tendencies_cuda.jl#L49 https://github.com/robertstrauss/MPAS_Ocean_Julia/blob/main/...
- Bromeo 3y ago
  Probably they could get a nice speed up by changing the if condition from your link into a multiplication with (eoe != 0) two lines down instead.
  - qrios 3y ago
    Good point! For me the beginner's mistake is: on a CPU any 'if' will cost you a clock cycle (branch prediction not considered), on a GPU it costs you the sync. In best case the syncs will come back over many loops, more often some cores will go out of sync more and more, eventually all other cores will have to wait for the last. Otherwise memory accesses will come blocking and that would cost all cores thousands of cycles.
marmalade2413 3y ago
The key elements that I consistently see missing when comparing languages is the cost/effort of maintenance and how easy is to write fast/slow code. The former boils down to, there's little point saving time in the writing of new code when new versions of the language hinder the use of old codes. Julia, being a very young language, simply does not have the legacy of backwards compatibility that Fortran has and as a result libraries written today are unlikely to work in 10 years time. Partly because of Julia's age and partly because backwards compatibility is not valued as much as it is in Fortran. The GNU Fortran compiler can still compile code written nearly 50 years ago (https://gcc.gnu.org/fortran/ https://gcc.gnu.org/fortran/). The second point is that benchmarks like this seem to focus on tuned implementations, which is really important, but another important aspect is how easy is it write fast code. With Fortran, it is very easy to write very fast code as a lot is abstracted away from the programmer so they can focus on FORmula TRANslation. My experience with Julia has not been the same and thus for people don't have the time to perform such optimisations, Fortran may still be the better choice. I do look forward to watching the development of Julia but for now I'll stick with using Fortran as the high performance numerical language in my tech stack. I'll finish with something I've read but I can't remember from where: Over the years there have been many examples of "X with replace Fortran" and the only thing that ends up getting replaced is X.

95 points by npalli 3y ago

TL;DR - Julia nice for prototyping - but zero speed up from using GPU - sometimes faster sometimes slower on CPU - no plans to use at the moment

oivey 3y ago
This is a little reductive. The speed comparison was between the highly parallel and optimized CPU version vs GPU. The GPU’s advertised FLOPS was only 4x the CPU setup’s. That the performance is so similar isn’t all that surprising and could come down to low level optimization details. There are likely some areas in Julia’s GPU stack that could be made better, but it’s certainly already pretty performant. It’s also comparing against Fortran. Their Python comparison is 100x+ slower. That a high level language even comes close is impressive.
- jppittma 3y ago
  If only they didn’t chose 1 based array indexing, like matlab. Having worked with matlab and for that reason I can’t seem to take them seriously.
  - baggepinnen 3y ago
    But you can take Fortran seriously?
  - pjmlp 3y ago
    Plenty of serious languages have either 1 based indexing, or allow the flexibility to set lower boundary explicitly.
  - marmalade2413 3y ago
    The choice of 1 based array indexing is due to Julia having it's roots in mathematics - which is a common choice for mathematics based programming languages (e.g. Fortran, Julia, MATLAB/GNU Octave, Mathematica, R and Wolfram). The reason being is that matrix notation starts at 1 and not 0. For a comparison on array indexing in different programming languages see here: https://en.wikipedia.org/wiki/Comparison_of_programming_languages_(array)#Array_dimensions https://en.wikipedia.org/wiki/Comparison_of_programming_lang... From personal experience, I wouldn't let something as simple as starting index getting in the way of programming for a language as you'll soon start automatically switching to thinking in 1 based and 0 based indexing depending on the programming language.
  - amj7e 3y ago
    If you think 0 or 1 based indexing will hinder your ability to write a program, it's not the indexing, it's a skill issue.
- pjmlp 3y ago
  Why should it be impressive? Compiled high level languages have always been pretty fast in general, for decades, before 20 years ago it started to be trendy to use interpreted scripting languages to do their work. Anyone used to Modula-2, Object Pascal, QuickBasic, Clipper, AMOS, Lisp, Scheme, Dylan, Smalltalk, SELF, Oberon, VB, Delphi, C++, SML, Miranda, Haskell, Prolog from 1980-1990's, isn't going to be surprised about speedups over Python.
  - oivey 3y ago
    100x relative to NumPy is still quite a lot, depending on how much of their code manages to stay in the C extension. If it was pure Python the delta would likely be much, much larger. Julia is definitely built on the shoulders of giants. I don’t have experience with most of those languages, especially in high performance Lisp or Haskell. Calling C++ high level is definitely not in the same vein as Python or Julia. My understanding is that Haskell can run into all sorts of performance gotchas because lazy evaluation can be unpredictable and recursion may not always easily unroll into optimizable loops. I think the combination of features, especially borrowed from Lisp and C++, really come together as something very productive for writing fast, very high level code.
    - pjmlp 3y ago
      C++ is definitely high level, in the same vein as Python and Julia, the only thing missing is reflection (if we ignore C++/CLI, C++ Builder and custom libraries), which with some luck might still make the cut to C++26. Just because some people insist in using it like C, doesn't make it less higher level in available features and tooling. Regardless of how easy it is to optimize Haskell code, even stupid Haskell code beats an interpreted language like Python. And C bindings are not Python, regardless of how the Python community likes to think otherwise. If anything, I look forward to Julia and others to put pressure in the community to take PyPy and other JIT attempts more seriously. Or maybe it doesn't matter and in a couple of years everyone has moved into mojo, if it is successful.
      - oivey 3y ago

npalli 3y ago

Ok, so a not very representative tl;dr was provided so adding mine here. 1. Serial CPU performance of Julia-unoptimized was around 12 times that of Python/Numpy. 2. Serial CPU performance of Julia optimized gave another 10-12x improvement leading to total 140x improvement over Python/Numpy. (good pointers on speeding up the Python code beyond Numpy listed but not tried or benchmarked) 3. Even though the GPU was rated for a listed performance of 9.7TFlops (A100) the code performed around the same as the optimized Serial CPU which has a listed performance of 2.5TFlops over 64 cores. Surprising result, there are some pointers to why that might be. 4. When comparing Julia-MPI with Fortran-MPI, at lower processor counts both match for total time, at middle processor counts (16-128) Fortran takes more time than Julia, at higher processor counts Julia becomes slower than Fortran. However the scaling down of Julia is driven more by a less efficient implementation of the communication time with the higher processor setup. Computation time scales well for both Julia and Fortran but the communication time takes over in the Julia case. Some nice optimization tips are provided to speed up Julia code. No plans to use Julia yet but it is good for prototyping and Fortran is good enough for now.

noobermin 3y ago
Just to be clear, doing well compared to an establish fortran code is an achievement. An established code is THE reference point.
- npalli 3y ago
  Yes, definitely very impressive results. Especially considering how much of an improvement over the Python/Numpy code they could achieve. If you are writing something from scratch, very good reason to work in Julia.
  - guenthert 3y ago
    Uhm, the poor performance of the Python/Numpy code indicates that there's something amiss; perhaps an apple/orange comparison. Don't get me wrong, Python is slow as molasses, but for this kind of applications, almost all time spent is (ought to be) spent in the Numpy code, which is mostly C/C++. Other systems might perform better, but I don't expect more than a factor of two (which itself would be huge in the HPC domain). Worth having a closer look methinks.

qrios 3y ago

For sure another case for the "you doing it wrong" team at Nvidia. But the decision to add Julia/CUDA.jl here was not the best idea on a comparison with really interesting insights about Fortran vs. Julia. The CUDA part of the code is nearly a one-to-one copy of the MPI code. Especially one sentence[1] (there are some similar, comparable blocks) explains why the code not performs very well. [1] https://github.com/robertstrauss/MPAS_Ocean_Julia/blob/main/mode_forward/calculate_tendencies_cuda.jl#L49 https://github.com/robertstrauss/MPAS_Ocean_Julia/blob/main/...

Bromeo 3y ago
Probably they could get a nice speed up by changing the if condition from your link into a multiplication with (eoe != 0) two lines down instead.
- qrios 3y ago
  Good point! For me the beginner's mistake is: on a CPU any 'if' will cost you a clock cycle (branch prediction not considered), on a GPU it costs you the sync. In best case the syncs will come back over many loops, more often some cores will go out of sync more and more, eventually all other cores will have to wait for the last. Otherwise memory accesses will come blocking and that would cost all cores thousands of cycles.

marmalade2413 3y ago

The key elements that I consistently see missing when comparing languages is the cost/effort of maintenance and how easy is to write fast/slow code. The former boils down to, there's little point saving time in the writing of new code when new versions of the language hinder the use of old codes. Julia, being a very young language, simply does not have the legacy of backwards compatibility that Fortran has and as a result libraries written today are unlikely to work in 10 years time. Partly because of Julia's age and partly because backwards compatibility is not valued as much as it is in Fortran. The GNU Fortran compiler can still compile code written nearly 50 years ago (https://gcc.gnu.org/fortran/ https://gcc.gnu.org/fortran/). The second point is that benchmarks like this seem to focus on tuned implementations, which is really important, but another important aspect is how easy is it write fast code. With Fortran, it is very easy to write very fast code as a lot is abstracted away from the programmer so they can focus on FORmula TRANslation. My experience with Julia has not been the same and thus for people don't have the time to perform such optimisations, Fortran may still be the better choice. I do look forward to watching the development of Julia but for now I'll stick with using Fortran as the high performance numerical language in my tech stack. I'll finish with something I've read but I can't remember from where: Over the years there have been many examples of "X with replace Fortran" and the only thing that ends up getting replaced is X.