8 ms·

Because C is slower than C. That is, they are compiling C with GCC, but Rust uses LLVM as a backend. Compiling their C code with clang reveals that it is also

Because C is slower than C. That is, they are compiling C with GCC, but Rust uses LLVM as a backend. Compiling their C code with clang reveals that it is also slightly slower than the C code compiled by GCC, but not faster than Rust. The actual conclusion here is that the GCC toolchain is slightly faster than the LLVM toolchain for this particular use case, but that isn't really news - it happens all the time. If one want to focus this on Rust, then the downside of Rust vs C here is that C has a GCC toolchain that happens to be faster for this use case, but Rust does not.

emmericp 7y ago
Thank you for your thoughtful comment, recommended reading: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html (edit: thanks for editing your comment, we can have a civil discussion here) To answer your question: Yes, that particular benchmark was done with GCC and yes, it would have been better to use clang/LLVM for a more fair comparison. We of course also compared gcc with clang/LLVM and found that clang was 0.8% slower, so it does not explain the difference entirely.
- fluffything 7y ago
  > We of course also compared gcc with clang/LLVM and found that clang was 0.8% slower, so it does not explain the difference entirely. I think it would be quite useful and interesting to figure out why there is still a small difference between clang and Rust. There might be some low hanging fruit in how Rust generates LLVM-IR that could be fixed.
  - emmericp 7y ago
    Another thing to point out: our 6% - 11% difference is already quite low. Netbricks [1] has a similar comparison between a Rust and a C network function (only the NF, driver in C in both cases) and they find 14% (LPM) to 20% (synthetic NF) difference. Fun fact: we've a system where Rust is faster than C despite still using more instructions, so yeah, neither instructions nor cycles tell the whole story... [1] https://people.eecs.berkeley.edu/~apanda/assets/papers/osdi16.pdf https://people.eecs.berkeley.edu/~apanda/assets/papers/osdi1...
    - kbenson 7y ago
      > Fun fact: we've a system where Rust is faster than C despite still using more instructions I think Bryan Cantrill did a pretty good explanation on differences you might see when rewriting something in Rust[1], and one of the things he looked at to see what was going on was Cycles Per Instruction. Instruction count itself means little if the instructions themselves have very different performance profiles and require different amounts of cycles to complete. Edit: You are tracking and reporting that, so it's not like I'm telling you anything you don't know. I still think the included article is well worth reading though. 1: http://dtrace.org/blogs/bmc/2018/09/28/the-relative-performance-of-c-and-rust/ http://dtrace.org/blogs/bmc/2018/09/28/the-relative-performa...
      - StreamBright 7y ago
        Thanks for sharing this, I haven't seen this video before but it i mind blowing.
      - StavrosK 7y ago
        Not only that, but your tight loop that takes up most of the execution time might have fewer instructions, whereas the rest of the program might have more, making the overall total higher but "instructions per time spent in function" lower.
    - fluffything 7y ago
      Since the resources of that paper are not available anymore, do you happen to know if they used Clang for compiling their C code ? A 10% perf difference between LLVM and GCC for different applications is in the order of magnitude of what all the hundreds of Phoronix benchmarks show every time a new version of these toolchains is released for general applications (e.g. not for micro-kernels).
  - ynx 7y ago
    doesn't it also use an older fork of LLVM rather than branching/rebasing off of master? That could account for losing out on optimizations.
    - fluffything 7y ago
      The Rust toolchain distributed by the Rust project bundles an LLVM version that's very close to LLVM master. IIRC, this version has some patches on top to allow Rust to query some backend information, but I'm not sure if this information is still up-to-date. Rust can, however, work with an external LLVM. When installing Rust through a package manager, e.g., on Linux, typical Linux distros like Debian, RH, Ubuntu, etc. configure Rust to use the system-wide LLVM binaries. So if you have Debian, and say LLVM 8.0 installed, you can just compare the installed clang 8.0 with the Rust version installed by the Linux distro. That will not compare the "bleeding edge" clang vs Rust toolchains, but it would be a fair comparison of Clang vs Rust at a particular point in those toolchains lifes.
- donio 7y ago
  It could be argued that using the best available implementation for each is the more fair comparison.
  - rejschaap 7y ago
    That depends on the purpose of the comparison. But for many practical reasons you are right. It makes this statement seem a bit dishonest though: "Our Rust driver is a few percent slower than our C driver, why? Well, it's of course because of safety features in Rust, but can we quantify that?"
  - fluffything 7y ago
    I think that comparing the best implementation of Rust vs the best implementation of C is an interesting thing to compare. But if that's what this post is comparing, most of the content is probably incorrect, because the main reason C binaries perform better is because the C backend of the C implementation used is better than the Rust backend _for this particular application_. This isn't really news. There are hundreds of benchmarks comparing C and C++ using the GCC and LLVM backends, and each is slightly better than the other, depending on the application. You don't even need to write code in two languages to show this. The authors appear to be aiming to compare how language differences affect the quality of the generated code. For that, you want to compare two equivalent-quality language frontends using an equivalent-quality optimization pipeline and code-generation backend. This is, in general, not feasible. But for all languages sharing the same LLVM or GCC backend, doing this comparison is trivial, and would limit the differences to either the frontend implementation, or the language itself.
013a 7y ago
Yeah, their initial explanation is that its the "safety features", but I was under the impression that literally everything related to rust's safety happens at compile time. Their big selling point is the "zero cost abstractions".
- atoav 7y ago
  While you are spot on with zero cost abstractions a bounds check in Rust is not without penalty (if enabled, should be off in release mode)
  - __s 7y ago
    Bounds checks are always on. Integer overflow checks are disabled in release mode. Bounds checks are necessary for memory safety, whereas integer overflow isn't
  - kjeetgill 7y ago
    I believe release mode keeps bound checks. It's integer overflow checks it gets rid of.
  - jnordwick 7y ago
    you can't turn bounds checking off. there are unsafe method for vec that allow you unchecked access to the array, but you need to to write it in unsafe block (some code that is performance sensitive does this).
- mehrdadn 7y ago
  I don't know... while it may not be the case here, I wouldn't be exactly shocked if the fact that Rust restricts your ability to do some things ends up having a performance penalty, even if any safety checks themselves happen at compile time.
- Zarel 7y ago
  Their initial explanation is bounds checking specifically. While most abstractions are zero-cost, bounds checking isn't. I do wish there were a way to track expectations about integer ranges in a type system, but there currently aren't.
paulddraper 7y ago
> the downside of Rust vs C here is that C has a GCC toolchain that happens to be faster for this use case, but Rust does not. Yes, this is the "legitimate" criticism to be made here. I say "legitimate" because LLVM has a bright future and I doubt many people see that as an actual issue.
m0zg 7y ago
That actually depends. I'm some cases I've seen Clang pull off superhuman feats of vectorization, especially on ARM64. You paste code in Godbolt Compiler Explorer and GCC produces fairly readable assembly, and Clang uses every trick in the book to vectorize. In those cases Clang can be substantially faster. But code needs to be written with vectorization in mind for that to happen (short loops of known size, and a few other restrictions like that). Most of the time it's a few percent behind GCC.
masklinn 7y ago
The very different load/store amounts, higher rate of instructions being retired and seemingly better L1 behaviour seem interesting though, does compile their C code under clang yield the same behaviour or is it closer to GCC's?
bjourne 7y ago
I similarly found that C was slightly faster than Rust in a microbenchmark called LPATHBench: https://gist.github.com/bjourne/4599a387d24c80906475b26b8ac9e5b7 https://gist.github.com/bjourne/4599a387d24c80906475b26b8ac9... That was with clang which performed significantly better than gcc. clang appears to generate very good code tree traversals. But this was way back in 2016. Numbers from microbenchmarks that old are hardly relevant anymore.

emmericp 7y ago

Thank you for your thoughtful comment, recommended reading: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html (edit: thanks for editing your comment, we can have a civil discussion here) To answer your question: Yes, that particular benchmark was done with GCC and yes, it would have been better to use clang/LLVM for a more fair comparison. We of course also compared gcc with clang/LLVM and found that clang was 0.8% slower, so it does not explain the difference entirely.

fluffything 7y ago
> We of course also compared gcc with clang/LLVM and found that clang was 0.8% slower, so it does not explain the difference entirely. I think it would be quite useful and interesting to figure out why there is still a small difference between clang and Rust. There might be some low hanging fruit in how Rust generates LLVM-IR that could be fixed.
- emmericp 7y ago
  Another thing to point out: our 6% - 11% difference is already quite low. Netbricks [1] has a similar comparison between a Rust and a C network function (only the NF, driver in C in both cases) and they find 14% (LPM) to 20% (synthetic NF) difference. Fun fact: we've a system where Rust is faster than C despite still using more instructions, so yeah, neither instructions nor cycles tell the whole story... [1] https://people.eecs.berkeley.edu/~apanda/assets/papers/osdi16.pdf https://people.eecs.berkeley.edu/~apanda/assets/papers/osdi1...
  - kbenson 7y ago
    > Fun fact: we've a system where Rust is faster than C despite still using more instructions I think Bryan Cantrill did a pretty good explanation on differences you might see when rewriting something in Rust[1], and one of the things he looked at to see what was going on was Cycles Per Instruction. Instruction count itself means little if the instructions themselves have very different performance profiles and require different amounts of cycles to complete. Edit: You are tracking and reporting that, so it's not like I'm telling you anything you don't know. I still think the included article is well worth reading though. 1: http://dtrace.org/blogs/bmc/2018/09/28/the-relative-performance-of-c-and-rust/ http://dtrace.org/blogs/bmc/2018/09/28/the-relative-performa...
    - StreamBright 7y ago
      Thanks for sharing this, I haven't seen this video before but it i mind blowing.
    - StavrosK 7y ago
      Not only that, but your tight loop that takes up most of the execution time might have fewer instructions, whereas the rest of the program might have more, making the overall total higher but "instructions per time spent in function" lower.
  - fluffything 7y ago
    Since the resources of that paper are not available anymore, do you happen to know if they used Clang for compiling their C code ? A 10% perf difference between LLVM and GCC for different applications is in the order of magnitude of what all the hundreds of Phoronix benchmarks show every time a new version of these toolchains is released for general applications (e.g. not for micro-kernels).
- ynx 7y ago
  doesn't it also use an older fork of LLVM rather than branching/rebasing off of master? That could account for losing out on optimizations.
  - fluffything 7y ago
    The Rust toolchain distributed by the Rust project bundles an LLVM version that's very close to LLVM master. IIRC, this version has some patches on top to allow Rust to query some backend information, but I'm not sure if this information is still up-to-date. Rust can, however, work with an external LLVM. When installing Rust through a package manager, e.g., on Linux, typical Linux distros like Debian, RH, Ubuntu, etc. configure Rust to use the system-wide LLVM binaries. So if you have Debian, and say LLVM 8.0 installed, you can just compare the installed clang 8.0 with the Rust version installed by the Linux distro. That will not compare the "bleeding edge" clang vs Rust toolchains, but it would be a fair comparison of Clang vs Rust at a particular point in those toolchains lifes.
donio 7y ago
It could be argued that using the best available implementation for each is the more fair comparison.
- rejschaap 7y ago
  That depends on the purpose of the comparison. But for many practical reasons you are right. It makes this statement seem a bit dishonest though: "Our Rust driver is a few percent slower than our C driver, why? Well, it's of course because of safety features in Rust, but can we quantify that?"
- fluffything 7y ago
  I think that comparing the best implementation of Rust vs the best implementation of C is an interesting thing to compare. But if that's what this post is comparing, most of the content is probably incorrect, because the main reason C binaries perform better is because the C backend of the C implementation used is better than the Rust backend _for this particular application_. This isn't really news. There are hundreds of benchmarks comparing C and C++ using the GCC and LLVM backends, and each is slightly better than the other, depending on the application. You don't even need to write code in two languages to show this. The authors appear to be aiming to compare how language differences affect the quality of the generated code. For that, you want to compare two equivalent-quality language frontends using an equivalent-quality optimization pipeline and code-generation backend. This is, in general, not feasible. But for all languages sharing the same LLVM or GCC backend, doing this comparison is trivial, and would limit the differences to either the frontend implementation, or the language itself.

013a 7y ago

Yeah, their initial explanation is that its the "safety features", but I was under the impression that literally everything related to rust's safety happens at compile time. Their big selling point is the "zero cost abstractions".

atoav 7y ago
While you are spot on with zero cost abstractions a bounds check in Rust is not without penalty (if enabled, should be off in release mode)
- __s 7y ago
  Bounds checks are always on. Integer overflow checks are disabled in release mode. Bounds checks are necessary for memory safety, whereas integer overflow isn't
- kjeetgill 7y ago
  I believe release mode keeps bound checks. It's integer overflow checks it gets rid of.
- jnordwick 7y ago
  you can't turn bounds checking off. there are unsafe method for vec that allow you unchecked access to the array, but you need to to write it in unsafe block (some code that is performance sensitive does this).
mehrdadn 7y ago
I don't know... while it may not be the case here, I wouldn't be exactly shocked if the fact that Rust restricts your ability to do some things ends up having a performance penalty, even if any safety checks themselves happen at compile time.
Zarel 7y ago
Their initial explanation is bounds checking specifically. While most abstractions are zero-cost, bounds checking isn't. I do wish there were a way to track expectations about integer ranges in a type system, but there currently aren't.

paulddraper 7y ago

> the downside of Rust vs C here is that C has a GCC toolchain that happens to be faster for this use case, but Rust does not. Yes, this is the "legitimate" criticism to be made here. I say "legitimate" because LLVM has a bright future and I doubt many people see that as an actual issue.

m0zg 7y ago

That actually depends. I'm some cases I've seen Clang pull off superhuman feats of vectorization, especially on ARM64. You paste code in Godbolt Compiler Explorer and GCC produces fairly readable assembly, and Clang uses every trick in the book to vectorize. In those cases Clang can be substantially faster. But code needs to be written with vectorization in mind for that to happen (short loops of known size, and a few other restrictions like that). Most of the time it's a few percent behind GCC.

masklinn 7y ago

The very different load/store amounts, higher rate of instructions being retired and seemingly better L1 behaviour seem interesting though, does compile their C code under clang yield the same behaviour or is it closer to GCC's?

bjourne 7y ago

I similarly found that C was slightly faster than Rust in a microbenchmark called LPATHBench: https://gist.github.com/bjourne/4599a387d24c80906475b26b8ac9e5b7 https://gist.github.com/bjourne/4599a387d24c80906475b26b8ac9... That was with clang which performed significantly better than gcc. clang appears to generate very good code tree traversals. But this was way back in 2016. Numbers from microbenchmarks that old are hardly relevant anymore.