19 ms·

Comparing Parallel Rust and C++

412 points by mmastrac 7y ago

sdan 7y ago
TLDR: Most of the time C++ with GCC was best.
- foota 7y ago
  Was it? Seemed to swap between rustc and gcc to me.
  - vihren 7y ago
    It seems that when they squeeze all possible optimizations, GCC was a bit faster, but for the not so optimized code GCC and rustc seem very comparable.
- maeln 7y ago
  Important to mention though: The difference was most of the time very slim.
- galangalalgol 7y ago
  Were you even looking at the same article? Rust was always better at intruction level parralellism and gcc was usually best at cache depending greatly on which i5 xeon was used. Clang was slways the worst.my main takeaway was that they are all three roughly identical and that small processor differences dominated the results.
  - fluffy87 7y ago
    Yeah, there were differences, but these were peanuts. Using two different versions of GCC will givenyou similar differences.
    - gpderetta 7y ago
      yes, you can see from the assembly listings in the final stages, that most of the differences were due to very minor code changes, due to details in the optimization passes and not really to language differences.
  - gpderetta 7y ago
    ILP is not the correct metric, total runtime is (I was confused as well initially, but it is well explained in the article). the initial rust code had higher ILP simply because it has to execute more code for the bound checks. The C++ code stalls waiting for memory, while rust can fill up those bubbles with the checks (and that's why the checks do not cause slowdown initially). But as the code gets progressively optimized, the stalls are removed and the the C++ code can match the rust ILP, but at this point rust needs to execute more code and get penalized leading to an higher runtime. After the bounds checks are removed from the rust code, both implementations converge to the same ILP and runtime.
  - Someone 7y ago
    "Clang was always the worst” Were you even looking at the same article? :-) I thought the same as you, until I looked at the results for the "Mid-range laptop CPU with 4 physical cores and 8 hardware threads.” There, clang does best in multi-core for v4 through v7. That made me think that, possibly, the default set of LLVM optimization transformation passes in clang is (better) optimised for Apple’s main audience: mobile. Also: why are the fastest run on the “4.3 GHz Mid-range desktop CPU” slower than the equivalent ones on the “3.4 GHz Mid-range laptop CPU”?
- tiborsaas 7y ago
  My TLDR: don't base your choice on performance if you have to pick between Rust or C++
GeekSquad13 7y ago
Geek squad agents who see all kinds of technical solutions like mobile smartphones. Geek squads save our device's external equipment and together they give it a new lease of life, And if you have to face any kind of problem, then our toll-free. Call on the Geek squad tech support number. https://geektechsupport.co.uk/ https://geektechsupport.co.uk/
ChrisSD 7y ago
Out of interest, why was lto (link time optimisation) set to false? I doubt this would affect the results much but it's useful for cross-crate inlining.
- OskarS 7y ago
  Presumably because the thing they're benchmarking is in a single translation unit, so LTO wouldn't matter. And it might screw up the benchmark if the C++ code was optimized between the "test harness" translation unit and the "code to benchmark" translation unit, but Rust wasn't. It's sensible for this kind of benchmark not to use LTO.
  - fluffy87 7y ago
    Not that it matters but you can use LTO with C++ and Rust, just need to compile and link both with LTO enabled.
gpderetta 7y ago
very nice. Most language comparison benchmark are completely useless other than for bragging points, but those, like this one, that go into details of why one specific implementation is faster or slower than another are much more interesting and allow making an idea of what makes a language slower or faster. Also, it interesting that the hand optimized program is about 100 times faster than the unoptimized one, showing that even today there is room for manual optimizations and you cannot trust the compiler blindly, but you have to iteratively work with it to get to an optimal solution. I can't figure out from either this article or the original one whether -fast-math was being used. Would be nice to know if that would help the compiler vectorize and unroll the loop with multiple accumulators.
- mratsim 7y ago
  If you are interested on the exact same topics (matrix multiplication parallelization) here are other step by step tutorials I used: - UlmBLAS [1], for a HPC course in 14 steps - BlisLab [2], make sure to checkout the tutorial.pdf. It gives you exercises to solve in C and each one build on the previous one In Rust matrixmultiply crate [3] implements those techniques to reach 80% of OpenBLAS speed, see blog post GEMM: a rabbit hole [4]. This is a generic approach that can be followed by any low-level languages. I reach 100% of OpenBLAS and MKL-DNN speed in Nim on large matrices as well [5] without any assembly and a generic code that can also generates integer matrix multiply [6]. Regarding fast-math, that's what you do manually, you interleave the fused-multiply adds as they have a latency of about 6 cycles (Broadwell, I don't remember on Skylake) [1]: http://apfel.mathematik.uni-ulm.de/~lehn/sghpc/gemm/ http://apfel.mathematik.uni-ulm.de/~lehn/sghpc/gemm/ [2]: https://github.com/flame/blislab https://github.com/flame/blislab [3]: https://github.com/bluss/matrixmultiply https://github.com/bluss/matrixmultiply [4]: https://bluss.github.io/rust/2016/03/28/a-gemmed-rabbit-hole/ https://bluss.github.io/rust/2016/03/28/a-gemmed-rabbit-hole... [5]: https://github.com/numforge/laser/blob/e660eeeb723426e80a7b1187864323d85527d18c/benchmarks/gemm/gemm_bench_float32.nim#L393-L439 https://github.com/numforge/laser/blob/e660eeeb723426e80a7b1... [6]: https://github.com/numforge/laser/blob/e660eeeb723426e80a7b1187864323d85527d18c/laser/primitives/matrix_multiplication/gemm_ukernel_avx512.nim https://github.com/numforge/laser/blob/e660eeeb723426e80a7b1...
keldaris 7y ago
This is a wonderfully written comparison benchmark and it deserves attention even for that reason alone. It knows its target audience, explains what's going on succinctly but completely, and avoids most of the usual benchmark pitfalls that result in comparing apples to oranges. Great job. The one glaring issue that is ignored is the floating point model used. I understand Rust still doesn't have a usable equivalent to -ffast-math, so I assume it wasn't used for that reason. Some discussion of whether it's permissible in this algorithm (I believe so?) and how much advantage that might give to C++ seems crucial when performance is a priority. Ironically, reading this has further convinced me that, for all of its disadvantages, I much prefer C++ to Rust for my needs. I'm sure others will draw the opposite conclusion and that's great. Rust is a language that clearly knows what it wants and if your priorities are aligned with that, the performance gap is shrinking and implementation-related reasons to avoid it are rapidly decreasing in number.
- mratsim 7y ago
  ffast-math is not needed when you unroll manually and write your own accumulators. See my benchmarks [1] with fast-math in the middle and the generated assembly comparison at the bottom. The base language is Nim but pure C code should generate the same assembly. [1] https://github.com/numforge/laser/blob/e660eeeb723426e80a7b1187864323d85527d18c/benchmarks/fp_reduction_latency/reduction_bench.nim#L304-L331 https://github.com/numforge/laser/blob/e660eeeb723426e80a7b1....
  - keldaris 7y ago
    That's absolutely right (and thank you for the reference), but this benchmark starts off with fairly naive code where -ffast-math may still help considerably.
    - mratsim 7y ago
      If in-between the start and the end, I went up to 8 accumulators and in files in the same folder I have the same with SSE and AVX instructions. I actually wanted to know exactly what ffast-math did, down to the codegen (so you have assembly at the bottom).
Hitton 7y ago
Awesome. I haven't started to learn Rust yet, but I still learned a lot.
kbumsik 7y ago
Maybe because I'm not familiar with Rust, but I always have kind of impressions that Rust code is very hard to read. It is just impossible to figure out what the code is doing for beginners. [C++ code]: https://github.com/parallel-rust-cpp/shortcut-comparison/blob/8cdab059d22eb8f30e1408c2fbf0ae666fa231d9/src/cpp/v7_cache_reuse/step.cpp https://github.com/parallel-rust-cpp/shortcut-comparison/blo... [Rust code]: https://github.com/parallel-rust-cpp/shortcut-comparison/blob/8cdab059d22eb8f30e1408c2fbf0ae666fa231d9/src/rust/v7_cache_reuse/src/lib.rs https://github.com/parallel-rust-cpp/shortcut-comparison/blo...
- xlc0212 7y ago
  How is C++ code better? Rust code is harder to write, but I don’t find it harder to read.
- _pferreir_ 7y ago
  It's not the most readable Rust code I've ever seen and I guess that's not the main concern here. That said, I guess people find C++ easier because they're already used to the syntax.
  - blub 7y ago
    Even an amateur C++ programmer or C programmer would understand what that C++ code does. The only slightly opaque thing is the std::tie call. I'd feel comfortable having one of my C-focused colleagues modifying that C++ code. The Rust code is... typical Rust. The algorithm almost gets lost in the specific peculiarities.
    - the_why_of_y 7y ago
      The first thing an amateur C++ programmer would notice is that that code isn't standard C++ code, what with its weird "#pragma omp" thing that isn't in the index of the Stroustrup book.
      - gpderetta 7y ago
fyp 7y ago
In case people missed the link, the reference implementation is an amazing post on its own: http://ppc.cs.aalto.fi/ch2/v7/ http://ppc.cs.aalto.fi/ch2/v7/
- xyst 7y ago
  So this is where my parallel computing professor got his material from. Definitely a good read for parallel computing, but that matrix multiplication problem was discussed to ad nauseum in the first few weeks.
wscott 7y ago
I am surprised he didn't benchmark the C++ program with Clang to give a closer comparison to rust. In my experience, despite all its other advantages, Clang still lags GCC a bit in raw performance. Still a really useful set of comparisons. I am impressed Rust is able to compete with all the magic OpenMP is doing in the background.
- steveklabnik 7y ago
  I haven’t read the whole thing yet, but clang seems to be benchmarked here: https://parallel-rust-cpp.github.io/results.html https://parallel-rust-cpp.github.io/results.html
- wscott 7y ago
  Wow, I totally misread that.
The_rationalist 7y ago
Would have been nice to compare code uglyness and performance of OpenMP 5 vs rayon + SIMD.
gatherhunterer 7y ago
For some reason many people are choosing to focus on code readability. This is not how a real-world program would be written. Who actually writes a single train-of-thought implementation without isolating concerns or using any code separation features? No realistic coder would expect their team to be satisfied working with code like this. It’s just a benchmark program, it doesn’t actually fit a use case or solve a problem.
- jcelerier 7y ago
  > This is not how a real-world program would be written. I have seen much more programs that look exactly like this (at least for the C++ part) than programs with clean separation of concerns in my life. > No realistic coder would expect their team to be satisfied working with code like this. assuming you've got a team (and that they are trained and not a bunch of interns or barely-graduated with 100% year-over-year turnover), assuming you are a professional developer and not someone who codes in the context of another profession (researcher, artist... heck, I've seen a music teacher making small python apps for the lobby of their music school once, etc).
ryuukk_ 7y ago
tldr; rust is slow
larusso 7y ago
Very cool. I love the visual explanations for certain iteration to understand the better see how the pre process step prepares the data. What I always miss in such posts is the full tool explanation how to retrieve the resulting assembly code. Don’t get me wrong I know how to search the internet. But the post goes quite some length to explain the basic rust setup. Maybe the post is aimed for veteran cpp programmers. I certainly would appreciate a link or example line how to generate the assembly code lines :)

412 points by mmastrac 7y ago

sdan 7y ago

TLDR: Most of the time C++ with GCC was best.

foota 7y ago
Was it? Seemed to swap between rustc and gcc to me.
- vihren 7y ago
  It seems that when they squeeze all possible optimizations, GCC was a bit faster, but for the not so optimized code GCC and rustc seem very comparable.
maeln 7y ago
Important to mention though: The difference was most of the time very slim.
galangalalgol 7y ago
Were you even looking at the same article? Rust was always better at intruction level parralellism and gcc was usually best at cache depending greatly on which i5 xeon was used. Clang was slways the worst.my main takeaway was that they are all three roughly identical and that small processor differences dominated the results.
- fluffy87 7y ago
  Yeah, there were differences, but these were peanuts. Using two different versions of GCC will givenyou similar differences.
  - gpderetta 7y ago
    yes, you can see from the assembly listings in the final stages, that most of the differences were due to very minor code changes, due to details in the optimization passes and not really to language differences.
- gpderetta 7y ago
  ILP is not the correct metric, total runtime is (I was confused as well initially, but it is well explained in the article). the initial rust code had higher ILP simply because it has to execute more code for the bound checks. The C++ code stalls waiting for memory, while rust can fill up those bubbles with the checks (and that's why the checks do not cause slowdown initially). But as the code gets progressively optimized, the stalls are removed and the the C++ code can match the rust ILP, but at this point rust needs to execute more code and get penalized leading to an higher runtime. After the bounds checks are removed from the rust code, both implementations converge to the same ILP and runtime.
- Someone 7y ago
  "Clang was always the worst” Were you even looking at the same article? :-) I thought the same as you, until I looked at the results for the "Mid-range laptop CPU with 4 physical cores and 8 hardware threads.” There, clang does best in multi-core for v4 through v7. That made me think that, possibly, the default set of LLVM optimization transformation passes in clang is (better) optimised for Apple’s main audience: mobile. Also: why are the fastest run on the “4.3 GHz Mid-range desktop CPU” slower than the equivalent ones on the “3.4 GHz Mid-range laptop CPU”?
tiborsaas 7y ago
My TLDR: don't base your choice on performance if you have to pick between Rust or C++

GeekSquad13 7y ago

Geek squad agents who see all kinds of technical solutions like mobile smartphones. Geek squads save our device's external equipment and together they give it a new lease of life, And if you have to face any kind of problem, then our toll-free. Call on the Geek squad tech support number. https://geektechsupport.co.uk/ https://geektechsupport.co.uk/

ChrisSD 7y ago

Out of interest, why was lto (link time optimisation) set to false? I doubt this would affect the results much but it's useful for cross-crate inlining.

OskarS 7y ago
Presumably because the thing they're benchmarking is in a single translation unit, so LTO wouldn't matter. And it might screw up the benchmark if the C++ code was optimized between the "test harness" translation unit and the "code to benchmark" translation unit, but Rust wasn't. It's sensible for this kind of benchmark not to use LTO.
- fluffy87 7y ago
  Not that it matters but you can use LTO with C++ and Rust, just need to compile and link both with LTO enabled.

gpderetta 7y ago

very nice. Most language comparison benchmark are completely useless other than for bragging points, but those, like this one, that go into details of why one specific implementation is faster or slower than another are much more interesting and allow making an idea of what makes a language slower or faster. Also, it interesting that the hand optimized program is about 100 times faster than the unoptimized one, showing that even today there is room for manual optimizations and you cannot trust the compiler blindly, but you have to iteratively work with it to get to an optimal solution. I can't figure out from either this article or the original one whether -fast-math was being used. Would be nice to know if that would help the compiler vectorize and unroll the loop with multiple accumulators.

mratsim 7y ago
If you are interested on the exact same topics (matrix multiplication parallelization) here are other step by step tutorials I used: - UlmBLAS [1], for a HPC course in 14 steps - BlisLab [2], make sure to checkout the tutorial.pdf. It gives you exercises to solve in C and each one build on the previous one In Rust matrixmultiply crate [3] implements those techniques to reach 80% of OpenBLAS speed, see blog post GEMM: a rabbit hole [4]. This is a generic approach that can be followed by any low-level languages. I reach 100% of OpenBLAS and MKL-DNN speed in Nim on large matrices as well [5] without any assembly and a generic code that can also generates integer matrix multiply [6]. Regarding fast-math, that's what you do manually, you interleave the fused-multiply adds as they have a latency of about 6 cycles (Broadwell, I don't remember on Skylake) [1]: http://apfel.mathematik.uni-ulm.de/~lehn/sghpc/gemm/ http://apfel.mathematik.uni-ulm.de/~lehn/sghpc/gemm/ [2]: https://github.com/flame/blislab https://github.com/flame/blislab [3]: https://github.com/bluss/matrixmultiply https://github.com/bluss/matrixmultiply [4]: https://bluss.github.io/rust/2016/03/28/a-gemmed-rabbit-hole/ https://bluss.github.io/rust/2016/03/28/a-gemmed-rabbit-hole... [5]: https://github.com/numforge/laser/blob/e660eeeb723426e80a7b1187864323d85527d18c/benchmarks/gemm/gemm_bench_float32.nim#L393-L439 https://github.com/numforge/laser/blob/e660eeeb723426e80a7b1... [6]: https://github.com/numforge/laser/blob/e660eeeb723426e80a7b1187864323d85527d18c/laser/primitives/matrix_multiplication/gemm_ukernel_avx512.nim https://github.com/numforge/laser/blob/e660eeeb723426e80a7b1...

keldaris 7y ago

This is a wonderfully written comparison benchmark and it deserves attention even for that reason alone. It knows its target audience, explains what's going on succinctly but completely, and avoids most of the usual benchmark pitfalls that result in comparing apples to oranges. Great job. The one glaring issue that is ignored is the floating point model used. I understand Rust still doesn't have a usable equivalent to -ffast-math, so I assume it wasn't used for that reason. Some discussion of whether it's permissible in this algorithm (I believe so?) and how much advantage that might give to C++ seems crucial when performance is a priority. Ironically, reading this has further convinced me that, for all of its disadvantages, I much prefer C++ to Rust for my needs. I'm sure others will draw the opposite conclusion and that's great. Rust is a language that clearly knows what it wants and if your priorities are aligned with that, the performance gap is shrinking and implementation-related reasons to avoid it are rapidly decreasing in number.

mratsim 7y ago
ffast-math is not needed when you unroll manually and write your own accumulators. See my benchmarks [1] with fast-math in the middle and the generated assembly comparison at the bottom. The base language is Nim but pure C code should generate the same assembly. [1] https://github.com/numforge/laser/blob/e660eeeb723426e80a7b1187864323d85527d18c/benchmarks/fp_reduction_latency/reduction_bench.nim#L304-L331 https://github.com/numforge/laser/blob/e660eeeb723426e80a7b1....
- keldaris 7y ago
  That's absolutely right (and thank you for the reference), but this benchmark starts off with fairly naive code where -ffast-math may still help considerably.
  - mratsim 7y ago
    If in-between the start and the end, I went up to 8 accumulators and in files in the same folder I have the same with SSE and AVX instructions. I actually wanted to know exactly what ffast-math did, down to the codegen (so you have assembly at the bottom).

Hitton 7y ago

Awesome. I haven't started to learn Rust yet, but I still learned a lot.

kbumsik 7y ago

Maybe because I'm not familiar with Rust, but I always have kind of impressions that Rust code is very hard to read. It is just impossible to figure out what the code is doing for beginners. [C++ code]: https://github.com/parallel-rust-cpp/shortcut-comparison/blob/8cdab059d22eb8f30e1408c2fbf0ae666fa231d9/src/cpp/v7_cache_reuse/step.cpp https://github.com/parallel-rust-cpp/shortcut-comparison/blo... [Rust code]: https://github.com/parallel-rust-cpp/shortcut-comparison/blob/8cdab059d22eb8f30e1408c2fbf0ae666fa231d9/src/rust/v7_cache_reuse/src/lib.rs https://github.com/parallel-rust-cpp/shortcut-comparison/blo...

xlc0212 7y ago
How is C++ code better? Rust code is harder to write, but I don’t find it harder to read.
_pferreir_ 7y ago
It's not the most readable Rust code I've ever seen and I guess that's not the main concern here. That said, I guess people find C++ easier because they're already used to the syntax.
- blub 7y ago
  Even an amateur C++ programmer or C programmer would understand what that C++ code does. The only slightly opaque thing is the std::tie call. I'd feel comfortable having one of my C-focused colleagues modifying that C++ code. The Rust code is... typical Rust. The algorithm almost gets lost in the specific peculiarities.
  - the_why_of_y 7y ago
    The first thing an amateur C++ programmer would notice is that that code isn't standard C++ code, what with its weird "#pragma omp" thing that isn't in the index of the Stroustrup book.
    - gpderetta 7y ago

fyp 7y ago

In case people missed the link, the reference implementation is an amazing post on its own: http://ppc.cs.aalto.fi/ch2/v7/ http://ppc.cs.aalto.fi/ch2/v7/

xyst 7y ago
So this is where my parallel computing professor got his material from. Definitely a good read for parallel computing, but that matrix multiplication problem was discussed to ad nauseum in the first few weeks.

wscott 7y ago

I am surprised he didn't benchmark the C++ program with Clang to give a closer comparison to rust. In my experience, despite all its other advantages, Clang still lags GCC a bit in raw performance. Still a really useful set of comparisons. I am impressed Rust is able to compete with all the magic OpenMP is doing in the background.

steveklabnik 7y ago
I haven’t read the whole thing yet, but clang seems to be benchmarked here: https://parallel-rust-cpp.github.io/results.html https://parallel-rust-cpp.github.io/results.html
wscott 7y ago
Wow, I totally misread that.

The_rationalist 7y ago

Would have been nice to compare code uglyness and performance of OpenMP 5 vs rayon + SIMD.

gatherhunterer 7y ago

For some reason many people are choosing to focus on code readability. This is not how a real-world program would be written. Who actually writes a single train-of-thought implementation without isolating concerns or using any code separation features? No realistic coder would expect their team to be satisfied working with code like this. It’s just a benchmark program, it doesn’t actually fit a use case or solve a problem.

jcelerier 7y ago
> This is not how a real-world program would be written. I have seen much more programs that look exactly like this (at least for the C++ part) than programs with clean separation of concerns in my life. > No realistic coder would expect their team to be satisfied working with code like this. assuming you've got a team (and that they are trained and not a bunch of interns or barely-graduated with 100% year-over-year turnover), assuming you are a professional developer and not someone who codes in the context of another profession (researcher, artist... heck, I've seen a music teacher making small python apps for the lobby of their music school once, etc).

ryuukk_ 7y ago

tldr; rust is slow

larusso 7y ago

Very cool. I love the visual explanations for certain iteration to understand the better see how the pre process step prepares the data. What I always miss in such posts is the full tool explanation how to retrieve the resulting assembly code. Don’t get me wrong I know how to search the internet. But the post goes quite some length to explain the basic rust setup. Maybe the post is aimed for veteran cpp programmers. I certainly would appreciate a link or example line how to generate the assembly code lines :)