3 ms·

> There is a lot of C++ code out there. A lot of it was not written by people who understand performance or who care. It is very easy to take that kind of code

> There is a lot of C++ code out there. A lot of it was not written by people who understand performance or who care. It is very easy to take that kind of code and make it faster, even without a switch to Rust. Cliff Biffle's great series of blog posts called "Learn Rust the Dangerous Way" takes the fastest C program from a benchmark game entry, rewrites it naïvely in Rust using copious amounts of `unsafe`, and then transforms it into program without `unsafe` while keeping it idiomatic. The last version is faster than the C implementation. It's a toy example but it's real-world example. http://cliffle.com/p/dangerust/ http://cliffle.com/p/dangerust/

pclmulqdq 4y ago
I'll read it and take a look. I am aware that Rust does very well on the benchmarks game. However, the benchmarks game (and microbenchmark-based comparisons in general) is hard to take seriously if you are thinking about application performance. Some languages microbenchmark very well, but don't translate that to system performance (C is the poster child of this effect), and some microbenchmark poorly but work very well in practical systems (Go is the most popular language with a big gap here, but some functional language like OCaml or Haskell probably has the biggest gap). The reasons for these gaps can include things like it being harder to use the optimal data structure for your application (eg C code using red-black trees instead of btrees in 2023) and large code size causing terrible caching behavior (heavily templated C++). I also remember seeing something here about some non-optimal calling convention in the Rust compiler, which would be another thing that shows up in a system that doesn't in a microbenchmark. Microbenchmarks are not a good replacement for system-level comparisons.
- igouy 4y ago
  > … hard to take seriously… Even when we're shown "Benchmarks are a crock" ? https://benchmarksgame-team.pages.debian.net/benchmarksgame/sometimes-people-just-make-up-stuff.html#benchmarks-are-a-crock https://benchmarksgame-team.pages.debian.net/benchmarksgame/... > Microbenchmarks are not a good replacement for system-level comparisons. For example — https://benchmarksgame-team.pages.debian.net/benchmarksgame/why-measure-toy-benchmark-programs.html#reimplement https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
  - pclmulqdq 4y ago
    I think we're in agreement here. Personally, I was really looking forward to seeing "we rewrote our database in Rust from C++ and QPS improved 5% because fast code was easier to write." This kind of benchmark is not BS, and actually would save you real money. Instead, the main arguments for the claim that Rust is the same speed as C++ are microbenchmarks, which actually are pretty useless when you are comparing very different implementations. The time they are useful is when you are refining an implementation. If you want to scrutinize the benchmarks game even further, they use gcc for their c compiler, where clang would probably be a better choice since the benchmarks are arithmetic-heavy.
    - nequo 4y ago
      When you get sufficiently close to the metal, the performance gains seem to be coming from better algorithms rather than from the programming language. Like you are saying: > The reasons for these gaps can include things like it being harder to use the optimal data structure for your application (eg C code using red-black trees instead of btrees in 2023) Bryan Cantrill described this experience[1] and maybe that is exactly what you are referring to. But if you're just looking for a language to get work done with, does it matter if Rust is faster because of better off-the-shelf data structures and algorithms or because of some inherent magic in the programming language? [1] See point 9 here: http://dtrace.org/blogs/bmc/2018/09/18/falling-in-love-with-rust/ http://dtrace.org/blogs/bmc/2018/09/18/falling-in-love-with-...
      - pclmulqdq 4y ago
        Yes, because my current alternative (modern C++ with Folly/Absl) provides those data structures already, so there's no real benefit to using the new thing.
        nequo 4y ago
        That's great for you! Then stick with your current setup. I don't already know C++. For me, learning Rust is easier than learning safe C++. I imagine that there are many people in a similar situation for whom Rust makes more sense than C++. That doesn't mean that it makes more sense for everyone.
- Shish2k 4y ago
  How big does a program need to be in order to stop counting as a microbenchmark?
  - igouy 4y ago
    https://benchmarksgame-team.pages.debian.net/benchmarksgame/why-measure-toy-benchmark-programs.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... "Milli benchmarks are not really hard Micro benchmarks are challenging, but OK Nano benchmarks are the damned beasts!" Slide 19 https://shipilev.net/talks/devoxx-Nov2013-benchmarking.pdf https://shipilev.net/talks/devoxx-Nov2013-benchmarking.pdf
  - pclmulqdq 4y ago
    Enough to stop performance slippage due to the increased amount of code and context. Unfortunately, the unit size is intentionally vague. For a lot of HPC applications, it's a small unit since they do the same math over and over again on large vectors (effectively the same as the microbenchmark). For databases and programs with significant business logic, it's a huge unit, almost a full-system test. For the companies that re-wrote their databases, login systems, and other similar things in Rust, a real benchmark comparison would be pretty easy, and they probably did it internally anyway. Hook one of your servers up to an artificial load generator and see how much it can take.
    - bmarkovic 4y ago
      Few years back, Convey[1] has apparently outran HAProxy in an alleged benchmark by the author[2]. That's a one man project (now abandoned, sadly) outrunning a decade old product built by an enterprise company AND a big community AND spearheaded and designed by a data structure genius. Granted, only in one of many tricks HAProxy can pull, but still. Not a database but indeed a concurrent world-facing RealWork software. If true (didn't actually check myself), I'd say it fits your bill. Personally, I read that as "can be as fast as, but without you having to be Willy Tarreau level genius" which is all I need. It's easier/more-intuitive to do a lot of things in C++, but safe, high performing C++ is certainly harder than safe, high performing Rust for huge swaths of use-cases. Also, as has been mentioned, its type system that benefitted from the PL research since the 80s also allows for nicer expression of business logic. In particular, this means that in Rust, unlike C, Go, or even C++ in great part, you are not writing in the same low-level intricate language at every level of your stack i.e. it can be a nicer high-level experience the higher you go if you designed your lower tiers well. And that last thing to me is the biggest advantage it has over the competition. Off course, there is also the fact that juggling dependencies in a non-trivial C++ project was a nightmare until recently with vcpkg and it's manifest mode and that will take probably another decade to become commonplace in the ecosystem (if ever). [1]: https://github.com/bparli/convey https://github.com/bparli/convey [2]: https://bparli.medium.com/adventures-in-rust-and-load-balancers-73a0bc61a192 https://bparli.medium.com/adventures-in-rust-and-load-balanc...
igouy 4y ago
Also gcc #9 n-body program was converted to Rust #9 — https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/nbody.html#intrinsics https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
- dralley 4y ago
  You've linked to an implementation full of hand-written SIMD, I'm not sure that's a great comparison one way or the other. Which is my biggest gripe with benchmark games. There really ought to be strict categories for "straightforward, idiomatic code someone with 3 YoE could write", "optimized-but-still maintainable code an experienced senior engineer would write", and "unrestricted wizardry".
  - igouy 4y ago
    Please provide a magic trick that will assign programs to "strict categories" in a way that no one will dispute :-) > … straightforward, idiomatic code… Even with tiny programs, once you ask for "idiomatic code" things stop being straightforward: different languages do things differently — https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/simple.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... > … hand-written SIMD… One approach is to filter those programs into their own section — hand-written vector instructions | "unsafe". Another is to use source code size as a proxy — https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/nbody-gz.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
  - nequo 4y ago
    It seems inherently difficult to define strict categories around the soft constraints that you are listing. But it would be interesting to try to approximate something like those categories. The project repo[1] contains the following call for idiomatic code: Please, people ask to see more "idiomatic" programs — - we already have enough exhaustively optimized Rust and C programs. - we already have enough hand-written vector SIMD and "unsafe" programs. Thank you. [1] https://salsa.debian.org/benchmarksgame-team/benchmarksgame https://salsa.debian.org/benchmarksgame-team/benchmarksgame

pclmulqdq 4y ago

I'll read it and take a look. I am aware that Rust does very well on the benchmarks game. However, the benchmarks game (and microbenchmark-based comparisons in general) is hard to take seriously if you are thinking about application performance. Some languages microbenchmark very well, but don't translate that to system performance (C is the poster child of this effect), and some microbenchmark poorly but work very well in practical systems (Go is the most popular language with a big gap here, but some functional language like OCaml or Haskell probably has the biggest gap). The reasons for these gaps can include things like it being harder to use the optimal data structure for your application (eg C code using red-black trees instead of btrees in 2023) and large code size causing terrible caching behavior (heavily templated C++). I also remember seeing something here about some non-optimal calling convention in the Rust compiler, which would be another thing that shows up in a system that doesn't in a microbenchmark. Microbenchmarks are not a good replacement for system-level comparisons.

igouy 4y ago
> … hard to take seriously… Even when we're shown "Benchmarks are a crock" ? https://benchmarksgame-team.pages.debian.net/benchmarksgame/sometimes-people-just-make-up-stuff.html#benchmarks-are-a-crock https://benchmarksgame-team.pages.debian.net/benchmarksgame/... > Microbenchmarks are not a good replacement for system-level comparisons. For example — https://benchmarksgame-team.pages.debian.net/benchmarksgame/why-measure-toy-benchmark-programs.html#reimplement https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
- pclmulqdq 4y ago
  I think we're in agreement here. Personally, I was really looking forward to seeing "we rewrote our database in Rust from C++ and QPS improved 5% because fast code was easier to write." This kind of benchmark is not BS, and actually would save you real money. Instead, the main arguments for the claim that Rust is the same speed as C++ are microbenchmarks, which actually are pretty useless when you are comparing very different implementations. The time they are useful is when you are refining an implementation. If you want to scrutinize the benchmarks game even further, they use gcc for their c compiler, where clang would probably be a better choice since the benchmarks are arithmetic-heavy.
  - nequo 4y ago
    When you get sufficiently close to the metal, the performance gains seem to be coming from better algorithms rather than from the programming language. Like you are saying: > The reasons for these gaps can include things like it being harder to use the optimal data structure for your application (eg C code using red-black trees instead of btrees in 2023) Bryan Cantrill described this experience[1] and maybe that is exactly what you are referring to. But if you're just looking for a language to get work done with, does it matter if Rust is faster because of better off-the-shelf data structures and algorithms or because of some inherent magic in the programming language? [1] See point 9 here: http://dtrace.org/blogs/bmc/2018/09/18/falling-in-love-with-rust/ http://dtrace.org/blogs/bmc/2018/09/18/falling-in-love-with-...
    - pclmulqdq 4y ago
      Yes, because my current alternative (modern C++ with Folly/Absl) provides those data structures already, so there's no real benefit to using the new thing.
      - nequo 4y ago
        That's great for you! Then stick with your current setup. I don't already know C++. For me, learning Rust is easier than learning safe C++. I imagine that there are many people in a similar situation for whom Rust makes more sense than C++. That doesn't mean that it makes more sense for everyone.
Shish2k 4y ago
How big does a program need to be in order to stop counting as a microbenchmark?
- igouy 4y ago
  https://benchmarksgame-team.pages.debian.net/benchmarksgame/why-measure-toy-benchmark-programs.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... "Milli benchmarks are not really hard Micro benchmarks are challenging, but OK Nano benchmarks are the damned beasts!" Slide 19 https://shipilev.net/talks/devoxx-Nov2013-benchmarking.pdf https://shipilev.net/talks/devoxx-Nov2013-benchmarking.pdf
- pclmulqdq 4y ago
  Enough to stop performance slippage due to the increased amount of code and context. Unfortunately, the unit size is intentionally vague. For a lot of HPC applications, it's a small unit since they do the same math over and over again on large vectors (effectively the same as the microbenchmark). For databases and programs with significant business logic, it's a huge unit, almost a full-system test. For the companies that re-wrote their databases, login systems, and other similar things in Rust, a real benchmark comparison would be pretty easy, and they probably did it internally anyway. Hook one of your servers up to an artificial load generator and see how much it can take.
  - bmarkovic 4y ago
    Few years back, Convey[1] has apparently outran HAProxy in an alleged benchmark by the author[2]. That's a one man project (now abandoned, sadly) outrunning a decade old product built by an enterprise company AND a big community AND spearheaded and designed by a data structure genius. Granted, only in one of many tricks HAProxy can pull, but still. Not a database but indeed a concurrent world-facing RealWork software. If true (didn't actually check myself), I'd say it fits your bill. Personally, I read that as "can be as fast as, but without you having to be Willy Tarreau level genius" which is all I need. It's easier/more-intuitive to do a lot of things in C++, but safe, high performing C++ is certainly harder than safe, high performing Rust for huge swaths of use-cases. Also, as has been mentioned, its type system that benefitted from the PL research since the 80s also allows for nicer expression of business logic. In particular, this means that in Rust, unlike C, Go, or even C++ in great part, you are not writing in the same low-level intricate language at every level of your stack i.e. it can be a nicer high-level experience the higher you go if you designed your lower tiers well. And that last thing to me is the biggest advantage it has over the competition. Off course, there is also the fact that juggling dependencies in a non-trivial C++ project was a nightmare until recently with vcpkg and it's manifest mode and that will take probably another decade to become commonplace in the ecosystem (if ever). [1]: https://github.com/bparli/convey https://github.com/bparli/convey [2]: https://bparli.medium.com/adventures-in-rust-and-load-balancers-73a0bc61a192 https://bparli.medium.com/adventures-in-rust-and-load-balanc...

igouy 4y ago

Also gcc #9 n-body program was converted to Rust #9 — https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/nbody.html#intrinsics https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

dralley 4y ago
You've linked to an implementation full of hand-written SIMD, I'm not sure that's a great comparison one way or the other. Which is my biggest gripe with benchmark games. There really ought to be strict categories for "straightforward, idiomatic code someone with 3 YoE could write", "optimized-but-still maintainable code an experienced senior engineer would write", and "unrestricted wizardry".
- igouy 4y ago
  Please provide a magic trick that will assign programs to "strict categories" in a way that no one will dispute :-) > … straightforward, idiomatic code… Even with tiny programs, once you ask for "idiomatic code" things stop being straightforward: different languages do things differently — https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/simple.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... > … hand-written SIMD… One approach is to filter those programs into their own section — hand-written vector instructions | "unsafe". Another is to use source code size as a proxy — https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/nbody-gz.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
- nequo 4y ago
  It seems inherently difficult to define strict categories around the soft constraints that you are listing. But it would be interesting to try to approximate something like those categories. The project repo[1] contains the following call for idiomatic code: Please, people ask to see more "idiomatic" programs — - we already have enough exhaustively optimized Rust and C programs. - we already have enough hand-written vector SIMD and "unsafe" programs. Thank you. [1] https://salsa.debian.org/benchmarksgame-team/benchmarksgame https://salsa.debian.org/benchmarksgame-team/benchmarksgame