4 ms·
I wrote that program in the hope that it would better illustrate why some of the benchmarks on the site aren't very good since for some benchmarks the program p
by JZerf 6y ago
I wrote that program in the hope that it would better illustrate why some of the benchmarks on the site aren't very good since for some benchmarks the program performance is highly dependent on the libraries being used and not the programming language implementation itself. I know at least one person opened an issue regarding this on the site issue tracker at https://salsa.debian.org/benchmarksgame-team/benchmarksgame/-/issues/317 https://salsa.debian.org/benchmarksgame-team/benchmarksgame/... and I also believe I recall one of the Rust regex crate developers mentioning this too.
I know that using PCRE2 in a Python program isn't typical but the Benchmarks Game does allow using other libraries and many of the previously submitted programs have been doing this for a long time already. The pidigits benchmark is the other benchmark that is heavily dependent on the libraries being used as is illustrated by their being a nearly ten way tie for second place by a bunch of programs that all use GMP.
- infoseek12 6y agoIt is an excellent illustration of that issue! And a needful reminder, as long as the implementations being compared are allowed use of that strategy. It would be interesting to see benchmark comparisons with somewhat more rigorous guidelines about what’s being compared, perhaps mandating the use of the same algorithm. That approach would introduce problems as well, like some languages benefiting more than others from the chosen algorithms. I think benchmarks are useful but the fact that they’re so often “flawed” or “misleading” may paradoxically mean that we need more of them so we can get a clearer view of the field.