4 ms·
Taking an average of wildly varying microbenchmark scores doesn't seem particularly useful. Furthermore Rust has never had particularly optimal implementations
by kibwen 11y ago
Taking an average of wildly varying microbenchmark scores doesn't seem particularly useful. Furthermore Rust has never had particularly optimal implementations for the benchmarks game (I can't speak as to the quality of the Go implementations). Furtherfurthermore this blog post is so old that the Rust results cannot be considered representative. Furtherfurtherfurthermore I would like it if we could just all collectively stop treating the benchmarks game as though any of its microbenchmarks measure anything other than "is this language capable of binding to libpcre" or "is this language capable of binding to libgmp". I swear, the benchmarks game is the SunSpider of programming language comparisons.
(Furtherfurtherfutherfurthermore, why are we still comparing Rust to Go? I thought we had gotten over this!)
- lucozade 11y agoAlthough you are of course correct, I would caution against a complete dismissal of micro benchmarks. Firstly because they are a sort of marketing: I've perused the shootout before and noticed an implementation get a higher score than I was expecting. This has sparked an interest in digging deeper into that implementation/language. Secondly because of the "can't even" problem i.e. why should I invest time investigating a new language if it can't even do simple benchmarks well? I appreciate that this isn't a particularly analytical view but I expect that it's not uncommon.
- kibwen 11y agoIt's a mistake to consider the benchmarks game a "can't even" problem, the level of effort given to each language's implementations is all over the map. This also isn't an utter dismissal of the concept of microbenchmarks, it's an utter dismissal of the concept of microbenchmarks as used to compare language implementations. Something like the TechEmpower web framework benchmarks is at least marginally better because the tasks permit a wider variation of implementations and at least pretend to do some kind of useful tasks, instead of pretending to be an arbiter of implementation quality (which, as much as the benchmarks game disclaims it, is still what people take it as).
- igouy 11y ago>>the level of effort given to each language's implementations is all over the map<< Evidently the level of effort to contribute a working Rust spectral-norm program is too high for anyone ;-) >>pretending to be an arbiter of implementation quality (which, as much as the benchmarks game disclaims it, is still what people take it as).<< The benchmarks game is "just 10 tiny examples."
- kibwen 11y ago> Evidently the level of effort to contribute a working > Rust spectral-norm program is too high for anyone ;-) Or they just don't care and refuse to let the benchmarks game hold their free time hostage. :P
- igouy 11y agoThat could be a respectable position; but not when "they" trash talk instead. That just looks bad.
- igouy 11y ago>>Taking an average of wildly varying microbenchmark scores doesn't seem particularly useful.<< Correct. Which is why, unlike Rohit Joshi, the benchmarks game simply shows a broad median; and when a summary measure is required to order the box plots, uses the geometric mean: http://benchmarksgame.alioth.debian.org/u64q/which-programs-are-fastest.html http://benchmarksgame.alioth.debian.org/u64q/which-programs-... (Note the reminders -- "Please don't obsess over tiny differences in median values from such a small number of examples." -- "Please don't obsess about which programming language implementation is shown 10th and which is shown 11th. You can see that the order would be different if it was based on the median scores instead of the geometric mean scores.") >>as though any of its microbenchmarks measure anything other than "is this language capable of binding to libpcre" or "is this language capable of binding to libgmp"<< Incorrect. Everyone can see that there is only 1 regex task and only 1 arbitrary precision arithmetic task -- and they sensibly allow library use.
- deleted 11y ago[deleted]