3 ms·

How big does a program need to be in order to stop counting as a microbenchmark?

igouy 4y ago
https://benchmarksgame-team.pages.debian.net/benchmarksgame/why-measure-toy-benchmark-programs.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... "Milli benchmarks are not really hard Micro benchmarks are challenging, but OK Nano benchmarks are the damned beasts!" Slide 19 https://shipilev.net/talks/devoxx-Nov2013-benchmarking.pdf https://shipilev.net/talks/devoxx-Nov2013-benchmarking.pdf
pclmulqdq 4y ago
Enough to stop performance slippage due to the increased amount of code and context. Unfortunately, the unit size is intentionally vague. For a lot of HPC applications, it's a small unit since they do the same math over and over again on large vectors (effectively the same as the microbenchmark). For databases and programs with significant business logic, it's a huge unit, almost a full-system test. For the companies that re-wrote their databases, login systems, and other similar things in Rust, a real benchmark comparison would be pretty easy, and they probably did it internally anyway. Hook one of your servers up to an artificial load generator and see how much it can take.
- bmarkovic 4y ago
  Few years back, Convey[1] has apparently outran HAProxy in an alleged benchmark by the author[2]. That's a one man project (now abandoned, sadly) outrunning a decade old product built by an enterprise company AND a big community AND spearheaded and designed by a data structure genius. Granted, only in one of many tricks HAProxy can pull, but still. Not a database but indeed a concurrent world-facing RealWork software. If true (didn't actually check myself), I'd say it fits your bill. Personally, I read that as "can be as fast as, but without you having to be Willy Tarreau level genius" which is all I need. It's easier/more-intuitive to do a lot of things in C++, but safe, high performing C++ is certainly harder than safe, high performing Rust for huge swaths of use-cases. Also, as has been mentioned, its type system that benefitted from the PL research since the 80s also allows for nicer expression of business logic. In particular, this means that in Rust, unlike C, Go, or even C++ in great part, you are not writing in the same low-level intricate language at every level of your stack i.e. it can be a nicer high-level experience the higher you go if you designed your lower tiers well. And that last thing to me is the biggest advantage it has over the competition. Off course, there is also the fact that juggling dependencies in a non-trivial C++ project was a nightmare until recently with vcpkg and it's manifest mode and that will take probably another decade to become commonplace in the ecosystem (if ever). [1]: https://github.com/bparli/convey https://github.com/bparli/convey [2]: https://bparli.medium.com/adventures-in-rust-and-load-balancers-73a0bc61a192 https://bparli.medium.com/adventures-in-rust-and-load-balanc...

https://benchmarksgame-team.pages.debian.net/benchmarksgame/why-measure-toy-benchmark-programs.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... "Milli benchmarks are not really hard Micro benchmarks are challenging, but OK Nano benchmarks are the damned beasts!" Slide 19 https://shipilev.net/talks/devoxx-Nov2013-benchmarking.pdf https://shipilev.net/talks/devoxx-Nov2013-benchmarking.pdf

pclmulqdq 4y ago

Enough to stop performance slippage due to the increased amount of code and context. Unfortunately, the unit size is intentionally vague. For a lot of HPC applications, it's a small unit since they do the same math over and over again on large vectors (effectively the same as the microbenchmark). For databases and programs with significant business logic, it's a huge unit, almost a full-system test. For the companies that re-wrote their databases, login systems, and other similar things in Rust, a real benchmark comparison would be pretty easy, and they probably did it internally anyway. Hook one of your servers up to an artificial load generator and see how much it can take.

bmarkovic 4y ago
Few years back, Convey[1] has apparently outran HAProxy in an alleged benchmark by the author[2]. That's a one man project (now abandoned, sadly) outrunning a decade old product built by an enterprise company AND a big community AND spearheaded and designed by a data structure genius. Granted, only in one of many tricks HAProxy can pull, but still. Not a database but indeed a concurrent world-facing RealWork software. If true (didn't actually check myself), I'd say it fits your bill. Personally, I read that as "can be as fast as, but without you having to be Willy Tarreau level genius" which is all I need. It's easier/more-intuitive to do a lot of things in C++, but safe, high performing C++ is certainly harder than safe, high performing Rust for huge swaths of use-cases. Also, as has been mentioned, its type system that benefitted from the PL research since the 80s also allows for nicer expression of business logic. In particular, this means that in Rust, unlike C, Go, or even C++ in great part, you are not writing in the same low-level intricate language at every level of your stack i.e. it can be a nicer high-level experience the higher you go if you designed your lower tiers well. And that last thing to me is the biggest advantage it has over the competition. Off course, there is also the fact that juggling dependencies in a non-trivial C++ project was a nightmare until recently with vcpkg and it's manifest mode and that will take probably another decade to become commonplace in the ecosystem (if ever). [1]: https://github.com/bparli/convey https://github.com/bparli/convey [2]: https://bparli.medium.com/adventures-in-rust-and-load-balancers-73a0bc61a192 https://bparli.medium.com/adventures-in-rust-and-load-balanc...