6 ms·

It's crazy for me to see that C is much slower than rust. I'm almost sure that there's something wrong there. In table 1 Rust Fortran C Go

by ktta 10y ago

It's crazy for me to see that C is much slower than rust. I'm almost sure that there's something wrong there. In table 1 Rust Fortran C Go 0m13.660s 0m14.640s 2m32.910s 4m26.240s They did have a note that C could be faster if language specific features could be used. Is rust really that performant compared to C? Or did they neglect to even bother checking C's performance?

reitzensteinm 10y ago
There's something wrong there. Either a corner case, or they are multithreading and vectorizing the Rust/Fortran but not the C. Which would be the equivalent of writing a comparison between a sledgehammer and jackhammer, without turning the latter on.
- marblestation 10y ago
  The code is available here: https://github.com/marblestation/benchmark-leapfrog https://github.com/marblestation/benchmark-leapfrog It would be great if somebody have a fresh look at the C version, since I completely agree that it is not normal that its execution time is so far away from Fortran or Rust.
  - walrus 10y ago
    On my system: gcc -O3 1m40.302s gcc -O3 -ffast-math 0m5.110s rustc -C opt-level=3 0m9.278s gcc (GCC) 6.3.1 20170109 rustc 1.15.1 (021bd294c 2017-02-08) Edit: Looking into this a bit more, I think 'rustc -C opt-level=3' is optimizing out the actual integration. If I put a println!("{}", x[0][0]) at the end, I end up with 1m43s. Not sure what 'gcc -O3 -ffast-math' is doing; I haven't looked at the disassembly.
    - marblestation 10y ago
      Fantastic, thanks!!!
    - simias 10y ago
      -ffast-math might be a bit cheating here, for scientific work it's not always applicable. Unless rustc uses fast math by default? That would be weird. It's surprising that gcc does such a poor job at -O3 though.
      - kibwen 10y ago
        Rust doesn't even have a way to turn on fastmath, AFAIK.
        dbaupp 10y ago
        It does, for individual operations: https://doc.rust-lang.org/std/?search=fast https://doc.rust-lang.org/std/?search=fast
    - tom_mellior 10y ago
      > If I put a println!("{}", x[0][0]) at the end, I end up with 1m43s. Yes, without that it's basically cheating. There is now a GitHub issue: https://github.com/marblestation/benchmark-leapfrog/issues/6 https://github.com/marblestation/benchmark-leapfrog/issues/6 The submitter writes: "final output [for Rust] is 134 seconds, and also the output is [[NaN, NaN, NaN], [NaN, NaN, NaN]] (batman)"
drivebyops 10y ago
Maybe that C code was the safe version which is inline with the rest?
timClicks 10y ago
It depends heavily on the (micro-)benchmark used. Rust is able to avoid allocations and de-allocations in many areas where C programmers would manually use alloc and free. I must admit though, that's less relevant to scientific programming, where the heaviest work is mainly matrix matrix multiplication and the sizes of everything is known in advance. Without reading through the paper, I expect work on the Rust/Fortran side is being vectorized and/or being kept in L1/L2 caches, whereas the C and Go compilers don't use SIMD instructions and/or are generating code that triggers cache misses.
cyber1 10y ago
I don't belive in this result! C slower than Rust in 11 times!!! Are they kidding?! http://benchmarksgame.alioth.debian.org/u64q/compare.php?lang=gcc&lang2=rust http://benchmarksgame.alioth.debian.org/u64q/compare.php?lan... -- in this benchmarks C in most cases faster than Rust, but only in few they have the same performance.
- Sean1708 10y ago
  Compiler optimisations can be very nuanced, especially in C. You might find that changing the code very slightly, or even just using a different combination of compiler options, can speed up the code a surprising amount. Edit: Looks like using -Ofast instead of -O3 makes a big difference: https://news.ycombinator.com/item?id=13634119 https://news.ycombinator.com/item?id=13634119
- bbatha 10y ago
  In most of those benchmarks rust is slower because it lacks SIMD support, not because of compiler optimizations.
  - kibwen 10y ago
    Or rather, Rust is slower on those because SIMD is currently only supported on nightly, and the benchmarks game chooses to use stable Rust exclusively.
    - pjmlp 10y ago
      I also don't work with VS Release Candidates for production code, as such I think it is faire to use only stable Rust.
      - dbaupp 10y ago
        From what I've seen, most of the Rust community agrees that using stable a perfectly fine choice; kibwen is offering context, not complaining.
marblestation 10y ago
We do not claim that Rust is so much performant than C, we just showed that Rust can be as fast as Fortran or C. And indeed, there must be something wrong with the C implementation. Pull requests with improvements are welcome: https://github.com/marblestation/benchmark-leapfrog https://github.com/marblestation/benchmark-leapfrog
- berkut 10y ago
  GCC produces almost 30% more instructions for the C version at -O3 level as opposed to -O2. Clang produces practically the same between the two levels. On OS X, so can't test if this actually makes a performance difference.
  - claudius 10y ago
    On my laptop (i7-5500U), I get the following timings (always with -march=native or -xHOST): GCC 6.3, -Ofast: 5.93s GCC 6.3, -O3: 93.06s GCC 6.3, -O2: 134.43s Clang 3.9, -O3: 346.92s Clang 3.9, -Ofast: 5.87s Intel 15.0, -O3: 110.84s Intel 15.0, -Ofast: 106.88s The Intel compiler is relatively old, but mostly it seems to be a case of IEEE 754 conformance -- if enabled, things take longer, if disabled, they’re (probably) as fast in C as in Rust.
  - eriknstr 10y ago
    >On OS X, so can't test if this actually makes a performance difference. The `time` command is part of base OS X.
    - Sean1708 10y ago
      But gcc is not.
      - eriknstr 10y ago
        In my reading of the comment it seemed to imply that the commenter already had both clang and gcc installed on their computer since they were talking about the amount of instructions produced by each. Perhaps I misunderstood?
panic 10y ago
For me, clang -O3 optimizes the main function into this single loop: LBB3_1: addsd %xmm1, %xmm0 addsd %xmm1, %xmm0 ucomisd %xmm0, %xmm2 jae LBB3_1 %xmm0 begins at 0, %xmm1 is 0.04, and %xmm2 is 3.6525E+8. In other words, it's measuring how long it takes to count to 3.6525E+8 by adding 0.04 repeatedly. LLVM (which both Rust and clang use as a backend) optimizes away the actual work of the program. GCC doesn't seem to optimize things out as aggressively, so that probably explains the difference: https://godbolt.org/g/lBRIIW https://godbolt.org/g/lBRIIW
- gpderetta 10y ago
  I.e the program has no side effects so it can be completely optimized away. Never trust a benchmark you didn't falsify yourself.
- michaf 10y ago
  gcc seems to do something similar for -Ofast. Adding something like printf("x[0][0]=%f\n", x[0][0]); at the end of main fixes this. Benchmarking is hard.
coldtea 10y ago
>It's crazy for me to see that C is much slower than rust. It's not in the general case, but can be for particular code. Unless the C program was written badly, the fact that Fortran is also faster seems to point to compiler optimizations not easy/possible in C (e.g. involving aliasing assurances).

by ktta 10y ago

reitzensteinm 10y ago

There's something wrong there. Either a corner case, or they are multithreading and vectorizing the Rust/Fortran but not the C. Which would be the equivalent of writing a comparison between a sledgehammer and jackhammer, without turning the latter on.

marblestation 10y ago
The code is available here: https://github.com/marblestation/benchmark-leapfrog https://github.com/marblestation/benchmark-leapfrog It would be great if somebody have a fresh look at the C version, since I completely agree that it is not normal that its execution time is so far away from Fortran or Rust.
- walrus 10y ago
  On my system: gcc -O3 1m40.302s gcc -O3 -ffast-math 0m5.110s rustc -C opt-level=3 0m9.278s gcc (GCC) 6.3.1 20170109 rustc 1.15.1 (021bd294c 2017-02-08) Edit: Looking into this a bit more, I think 'rustc -C opt-level=3' is optimizing out the actual integration. If I put a println!("{}", x[0][0]) at the end, I end up with 1m43s. Not sure what 'gcc -O3 -ffast-math' is doing; I haven't looked at the disassembly.
  - marblestation 10y ago
    Fantastic, thanks!!!
  - simias 10y ago
    -ffast-math might be a bit cheating here, for scientific work it's not always applicable. Unless rustc uses fast math by default? That would be weird. It's surprising that gcc does such a poor job at -O3 though.
    - kibwen 10y ago
      Rust doesn't even have a way to turn on fastmath, AFAIK.
      - dbaupp 10y ago
        It does, for individual operations: https://doc.rust-lang.org/std/?search=fast https://doc.rust-lang.org/std/?search=fast
  - tom_mellior 10y ago
    > If I put a println!("{}", x[0][0]) at the end, I end up with 1m43s. Yes, without that it's basically cheating. There is now a GitHub issue: https://github.com/marblestation/benchmark-leapfrog/issues/6 https://github.com/marblestation/benchmark-leapfrog/issues/6 The submitter writes: "final output [for Rust] is 134 seconds, and also the output is [[NaN, NaN, NaN], [NaN, NaN, NaN]] (batman)"

drivebyops 10y ago

Maybe that C code was the safe version which is inline with the rest?

timClicks 10y ago

It depends heavily on the (micro-)benchmark used. Rust is able to avoid allocations and de-allocations in many areas where C programmers would manually use alloc and free. I must admit though, that's less relevant to scientific programming, where the heaviest work is mainly matrix matrix multiplication and the sizes of everything is known in advance. Without reading through the paper, I expect work on the Rust/Fortran side is being vectorized and/or being kept in L1/L2 caches, whereas the C and Go compilers don't use SIMD instructions and/or are generating code that triggers cache misses.

cyber1 10y ago

I don't belive in this result! C slower than Rust in 11 times!!! Are they kidding?! http://benchmarksgame.alioth.debian.org/u64q/compare.php?lang=gcc&lang2=rust http://benchmarksgame.alioth.debian.org/u64q/compare.php?lan... -- in this benchmarks C in most cases faster than Rust, but only in few they have the same performance.

Sean1708 10y ago
Compiler optimisations can be very nuanced, especially in C. You might find that changing the code very slightly, or even just using a different combination of compiler options, can speed up the code a surprising amount. Edit: Looks like using -Ofast instead of -O3 makes a big difference: https://news.ycombinator.com/item?id=13634119 https://news.ycombinator.com/item?id=13634119
bbatha 10y ago
In most of those benchmarks rust is slower because it lacks SIMD support, not because of compiler optimizations.
- kibwen 10y ago
  Or rather, Rust is slower on those because SIMD is currently only supported on nightly, and the benchmarks game chooses to use stable Rust exclusively.
  - pjmlp 10y ago
    I also don't work with VS Release Candidates for production code, as such I think it is faire to use only stable Rust.
    - dbaupp 10y ago
      From what I've seen, most of the Rust community agrees that using stable a perfectly fine choice; kibwen is offering context, not complaining.

marblestation 10y ago

We do not claim that Rust is so much performant than C, we just showed that Rust can be as fast as Fortran or C. And indeed, there must be something wrong with the C implementation. Pull requests with improvements are welcome: https://github.com/marblestation/benchmark-leapfrog https://github.com/marblestation/benchmark-leapfrog

berkut 10y ago
GCC produces almost 30% more instructions for the C version at -O3 level as opposed to -O2. Clang produces practically the same between the two levels. On OS X, so can't test if this actually makes a performance difference.
- claudius 10y ago
  On my laptop (i7-5500U), I get the following timings (always with -march=native or -xHOST): GCC 6.3, -Ofast: 5.93s GCC 6.3, -O3: 93.06s GCC 6.3, -O2: 134.43s Clang 3.9, -O3: 346.92s Clang 3.9, -Ofast: 5.87s Intel 15.0, -O3: 110.84s Intel 15.0, -Ofast: 106.88s The Intel compiler is relatively old, but mostly it seems to be a case of IEEE 754 conformance -- if enabled, things take longer, if disabled, they’re (probably) as fast in C as in Rust.
- eriknstr 10y ago
  >On OS X, so can't test if this actually makes a performance difference. The `time` command is part of base OS X.
  - Sean1708 10y ago
    But gcc is not.
    - eriknstr 10y ago
      In my reading of the comment it seemed to imply that the commenter already had both clang and gcc installed on their computer since they were talking about the amount of instructions produced by each. Perhaps I misunderstood?

panic 10y ago

For me, clang -O3 optimizes the main function into this single loop: LBB3_1: addsd %xmm1, %xmm0 addsd %xmm1, %xmm0 ucomisd %xmm0, %xmm2 jae LBB3_1 %xmm0 begins at 0, %xmm1 is 0.04, and %xmm2 is 3.6525E+8. In other words, it's measuring how long it takes to count to 3.6525E+8 by adding 0.04 repeatedly. LLVM (which both Rust and clang use as a backend) optimizes away the actual work of the program. GCC doesn't seem to optimize things out as aggressively, so that probably explains the difference: https://godbolt.org/g/lBRIIW https://godbolt.org/g/lBRIIW

gpderetta 10y ago
I.e the program has no side effects so it can be completely optimized away. Never trust a benchmark you didn't falsify yourself.
michaf 10y ago
gcc seems to do something similar for -Ofast. Adding something like printf("x[0][0]=%f\n", x[0][0]); at the end of main fixes this. Benchmarking is hard.

coldtea 10y ago

>It's crazy for me to see that C is much slower than rust. It's not in the general case, but can be for particular code. Unless the C program was written badly, the fact that Fortran is also faster seems to point to compiler optimizations not easy/possible in C (e.g. involving aliasing assurances).