4 ms·
> The programs on the site fit the rules defined on it, and the rules include the verification of the results I'm not sure what you're refuting? Verifying tha
by kibwen 10y ago
> The programs on the site fit the rules defined on it, and the rules include the verification of the results
I'm not sure what you're refuting?
Verifying that the results match is a necessary but insufficient quality for ensuring comparability. If the algorithm could be the same in each language but isn't (e.g. quicksort vs. bogosort), then that's not a valid comparison if your objective is to determine the overhead imposed by the language implementation itself (and if you're not trying to determine language implementation overhead, then what are you measuring?). Likewise if the implementation details could be the same in each language but aren't (e.g. if one uses 64-bit integers and the other uses 32-bit integers).
The computer language benchmarks game was initially conceived to determine a ballpark for how slow interpreted and managed languages are compared to C. Quantifying the overhead of interpreters and runtimes is its raison d'etre, and it shows. When it comes to comparing low-level systems languages that have no runtime to speak of, the best it can do is attempt to quantify the quality of each backend's code generator (it's a missed opportunity that it doesn't include Clang for comparison with GCC).
(And yes, I understand that the benchmarks game contains repeated massive disclaimers that people should not take the performance results as a means of serious comparison. Internet commentators remain undeterred.)
If you're just trying to argue that the methodology used in the OP is poor, then obviously we're in agreement (was there ever any doubt?).
- igouy 10y ago> The computer language benchmarks game was initially conceived to determine a ballpark for how slow interpreted and managed languages are compared to C. Today the benchmarks game is referred to from the Rust FAQ. What is one to do? https://www.rust-lang.org/en-US/faq.html#performance https://www.rust-lang.org/en-US/faq.html#performance Back in the previous millenia -- "[Doug Bagley's] goal was to compare all the major scripting languages. Then [Doug Bagley] started adding in some compiled languages for comparison…" http://web.archive.org/web/20010125021400/http://www.bagley.org/~doug/shootout/#News http://web.archive.org/web/20010125021400/http://www.bagley....
- igouy 10y ago> …repeated massive disclaimers that people should not take the performance results as a means of serious comparison. Actually one small disclaimer -- "Non-motivation: We are profoundly uninterested in claims that these measurements, of a few tiny programs, somehow define the relative performance of programming languages." There's much more about PL benchmarks in general.