3 ms·

I'll read it and take a look. I am aware that Rust does very well on the benchmarks game. However, the benchmarks game (and microbenchmark-based comparisons i

I'll read it and take a look. I am aware that Rust does very well on the benchmarks game. However, the benchmarks game (and microbenchmark-based comparisons in general) is hard to take seriously if you are thinking about application performance. Some languages microbenchmark very well, but don't translate that to system performance (C is the poster child of this effect), and some microbenchmark poorly but work very well in practical systems (Go is the most popular language with a big gap here, but some functional language like OCaml or Haskell probably has the biggest gap). The reasons for these gaps can include things like it being harder to use the optimal data structure for your application (eg C code using red-black trees instead of btrees in 2023) and large code size causing terrible caching behavior (heavily templated C++). I also remember seeing something here about some non-optimal calling convention in the Rust compiler, which would be another thing that shows up in a system that doesn't in a microbenchmark. Microbenchmarks are not a good replacement for system-level comparisons.

igouy 4y ago
> … hard to take seriously… Even when we're shown "Benchmarks are a crock" ? https://benchmarksgame-team.pages.debian.net/benchmarksgame/sometimes-people-just-make-up-stuff.html#benchmarks-are-a-crock https://benchmarksgame-team.pages.debian.net/benchmarksgame/... > Microbenchmarks are not a good replacement for system-level comparisons. For example — https://benchmarksgame-team.pages.debian.net/benchmarksgame/why-measure-toy-benchmark-programs.html#reimplement https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
- pclmulqdq 4y ago
  I think we're in agreement here. Personally, I was really looking forward to seeing "we rewrote our database in Rust from C++ and QPS improved 5% because fast code was easier to write." This kind of benchmark is not BS, and actually would save you real money. Instead, the main arguments for the claim that Rust is the same speed as C++ are microbenchmarks, which actually are pretty useless when you are comparing very different implementations. The time they are useful is when you are refining an implementation. If you want to scrutinize the benchmarks game even further, they use gcc for their c compiler, where clang would probably be a better choice since the benchmarks are arithmetic-heavy.
  - nequo 4y ago
    When you get sufficiently close to the metal, the performance gains seem to be coming from better algorithms rather than from the programming language. Like you are saying: > The reasons for these gaps can include things like it being harder to use the optimal data structure for your application (eg C code using red-black trees instead of btrees in 2023) Bryan Cantrill described this experience[1] and maybe that is exactly what you are referring to. But if you're just looking for a language to get work done with, does it matter if Rust is faster because of better off-the-shelf data structures and algorithms or because of some inherent magic in the programming language? [1] See point 9 here: http://dtrace.org/blogs/bmc/2018/09/18/falling-in-love-with-rust/ http://dtrace.org/blogs/bmc/2018/09/18/falling-in-love-with-...
    - pclmulqdq 4y ago
      Yes, because my current alternative (modern C++ with Folly/Absl) provides those data structures already, so there's no real benefit to using the new thing.
      - nequo 4y ago
        That's great for you! Then stick with your current setup. I don't already know C++. For me, learning Rust is easier than learning safe C++. I imagine that there are many people in a similar situation for whom Rust makes more sense than C++. That doesn't mean that it makes more sense for everyone.
  - igouy 4y ago
    > This kind of benchmark is not BS, and actually would save you real money. And would cost someone else real money to do — so microbenchmarks ;-) > … they use gcc for their c compiler, where clang would probably be a better choice… https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/c.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
    - pclmulqdq 4y ago
Shish2k 4y ago
How big does a program need to be in order to stop counting as a microbenchmark?
- igouy 4y ago
  https://benchmarksgame-team.pages.debian.net/benchmarksgame/why-measure-toy-benchmark-programs.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... "Milli benchmarks are not really hard Micro benchmarks are challenging, but OK Nano benchmarks are the damned beasts!" Slide 19 https://shipilev.net/talks/devoxx-Nov2013-benchmarking.pdf https://shipilev.net/talks/devoxx-Nov2013-benchmarking.pdf
- pclmulqdq 4y ago
  Enough to stop performance slippage due to the increased amount of code and context. Unfortunately, the unit size is intentionally vague. For a lot of HPC applications, it's a small unit since they do the same math over and over again on large vectors (effectively the same as the microbenchmark). For databases and programs with significant business logic, it's a huge unit, almost a full-system test. For the companies that re-wrote their databases, login systems, and other similar things in Rust, a real benchmark comparison would be pretty easy, and they probably did it internally anyway. Hook one of your servers up to an artificial load generator and see how much it can take.
  - bmarkovic 4y ago
    Few years back, Convey[1] has apparently outran HAProxy in an alleged benchmark by the author[2]. That's a one man project (now abandoned, sadly) outrunning a decade old product built by an enterprise company AND a big community AND spearheaded and designed by a data structure genius. Granted, only in one of many tricks HAProxy can pull, but still. Not a database but indeed a concurrent world-facing RealWork software. If true (didn't actually check myself), I'd say it fits your bill. Personally, I read that as "can be as fast as, but without you having to be Willy Tarreau level genius" which is all I need. It's easier/more-intuitive to do a lot of things in C++, but safe, high performing C++ is certainly harder than safe, high performing Rust for huge swaths of use-cases. Also, as has been mentioned, its type system that benefitted from the PL research since the 80s also allows for nicer expression of business logic. In particular, this means that in Rust, unlike C, Go, or even C++ in great part, you are not writing in the same low-level intricate language at every level of your stack i.e. it can be a nicer high-level experience the higher you go if you designed your lower tiers well. And that last thing to me is the biggest advantage it has over the competition. Off course, there is also the fact that juggling dependencies in a non-trivial C++ project was a nightmare until recently with vcpkg and it's manifest mode and that will take probably another decade to become commonplace in the ecosystem (if ever). [1]: https://github.com/bparli/convey https://github.com/bparli/convey [2]: https://bparli.medium.com/adventures-in-rust-and-load-balancers-73a0bc61a192 https://bparli.medium.com/adventures-in-rust-and-load-balanc...

igouy 4y ago

> … hard to take seriously… Even when we're shown "Benchmarks are a crock" ? https://benchmarksgame-team.pages.debian.net/benchmarksgame/sometimes-people-just-make-up-stuff.html#benchmarks-are-a-crock https://benchmarksgame-team.pages.debian.net/benchmarksgame/... > Microbenchmarks are not a good replacement for system-level comparisons. For example — https://benchmarksgame-team.pages.debian.net/benchmarksgame/why-measure-toy-benchmark-programs.html#reimplement https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

pclmulqdq 4y ago
I think we're in agreement here. Personally, I was really looking forward to seeing "we rewrote our database in Rust from C++ and QPS improved 5% because fast code was easier to write." This kind of benchmark is not BS, and actually would save you real money. Instead, the main arguments for the claim that Rust is the same speed as C++ are microbenchmarks, which actually are pretty useless when you are comparing very different implementations. The time they are useful is when you are refining an implementation. If you want to scrutinize the benchmarks game even further, they use gcc for their c compiler, where clang would probably be a better choice since the benchmarks are arithmetic-heavy.
- nequo 4y ago
  When you get sufficiently close to the metal, the performance gains seem to be coming from better algorithms rather than from the programming language. Like you are saying: > The reasons for these gaps can include things like it being harder to use the optimal data structure for your application (eg C code using red-black trees instead of btrees in 2023) Bryan Cantrill described this experience[1] and maybe that is exactly what you are referring to. But if you're just looking for a language to get work done with, does it matter if Rust is faster because of better off-the-shelf data structures and algorithms or because of some inherent magic in the programming language? [1] See point 9 here: http://dtrace.org/blogs/bmc/2018/09/18/falling-in-love-with-rust/ http://dtrace.org/blogs/bmc/2018/09/18/falling-in-love-with-...
  - pclmulqdq 4y ago
    Yes, because my current alternative (modern C++ with Folly/Absl) provides those data structures already, so there's no real benefit to using the new thing.
    - nequo 4y ago
      That's great for you! Then stick with your current setup. I don't already know C++. For me, learning Rust is easier than learning safe C++. I imagine that there are many people in a similar situation for whom Rust makes more sense than C++. That doesn't mean that it makes more sense for everyone.
- igouy 4y ago
  > This kind of benchmark is not BS, and actually would save you real money. And would cost someone else real money to do — so microbenchmarks ;-) > … they use gcc for their c compiler, where clang would probably be a better choice… https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/c.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
  - pclmulqdq 4y ago

Shish2k 4y ago

How big does a program need to be in order to stop counting as a microbenchmark?

igouy 4y ago
https://benchmarksgame-team.pages.debian.net/benchmarksgame/why-measure-toy-benchmark-programs.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... "Milli benchmarks are not really hard Micro benchmarks are challenging, but OK Nano benchmarks are the damned beasts!" Slide 19 https://shipilev.net/talks/devoxx-Nov2013-benchmarking.pdf https://shipilev.net/talks/devoxx-Nov2013-benchmarking.pdf
pclmulqdq 4y ago
Enough to stop performance slippage due to the increased amount of code and context. Unfortunately, the unit size is intentionally vague. For a lot of HPC applications, it's a small unit since they do the same math over and over again on large vectors (effectively the same as the microbenchmark). For databases and programs with significant business logic, it's a huge unit, almost a full-system test. For the companies that re-wrote their databases, login systems, and other similar things in Rust, a real benchmark comparison would be pretty easy, and they probably did it internally anyway. Hook one of your servers up to an artificial load generator and see how much it can take.
- bmarkovic 4y ago
  Few years back, Convey[1] has apparently outran HAProxy in an alleged benchmark by the author[2]. That's a one man project (now abandoned, sadly) outrunning a decade old product built by an enterprise company AND a big community AND spearheaded and designed by a data structure genius. Granted, only in one of many tricks HAProxy can pull, but still. Not a database but indeed a concurrent world-facing RealWork software. If true (didn't actually check myself), I'd say it fits your bill. Personally, I read that as "can be as fast as, but without you having to be Willy Tarreau level genius" which is all I need. It's easier/more-intuitive to do a lot of things in C++, but safe, high performing C++ is certainly harder than safe, high performing Rust for huge swaths of use-cases. Also, as has been mentioned, its type system that benefitted from the PL research since the 80s also allows for nicer expression of business logic. In particular, this means that in Rust, unlike C, Go, or even C++ in great part, you are not writing in the same low-level intricate language at every level of your stack i.e. it can be a nicer high-level experience the higher you go if you designed your lower tiers well. And that last thing to me is the biggest advantage it has over the competition. Off course, there is also the fact that juggling dependencies in a non-trivial C++ project was a nightmare until recently with vcpkg and it's manifest mode and that will take probably another decade to become commonplace in the ecosystem (if ever). [1]: https://github.com/bparli/convey https://github.com/bparli/convey [2]: https://bparli.medium.com/adventures-in-rust-and-load-balancers-73a0bc61a192 https://bparli.medium.com/adventures-in-rust-and-load-balanc...