4 ms·

The programs on the site fit the rules defined on it, and the rules include the verification of the results: https://benchmarksgame.alioth.debian.org/why-measu

by acqq 10y ago

The programs on the site fit the rules defined on it, and the rules include the verification of the results: https://benchmarksgame.alioth.debian.org/why-measure-toy-benchmark-programs.html https://benchmarksgame.alioth.debian.org/why-measure-toy-ben... The OP doesn't even specify the rules for its own benchmark, as far as I understand doesn't verify the results? People here get NaNs? If the NaNs are produced as results, then the OP code is not measuring the speed of calculations at all but the speed of the failed calculations. Which is especially problematic as the main argument is "attractive for the scientific community" "it guarantees memory safety." Which is presented as good because not having it "can produce random behaviors and affect the scientific interpretation of the results." If the result here is NaN the calculation doesn't even have to be performed after the first NaN that affects the result appears, as anything + NaN is NaN etc. Edit: kibwen, did I write somewhere that I "refute" something? I gave a link to the rationale behind the "benchmarksgame" site. The rest is about the OP, surely not about your post.

kibwen 10y ago
> The programs on the site fit the rules defined on it, and the rules include the verification of the results I'm not sure what you're refuting? Verifying that the results match is a necessary but insufficient quality for ensuring comparability. If the algorithm could be the same in each language but isn't (e.g. quicksort vs. bogosort), then that's not a valid comparison if your objective is to determine the overhead imposed by the language implementation itself (and if you're not trying to determine language implementation overhead, then what are you measuring?). Likewise if the implementation details could be the same in each language but aren't (e.g. if one uses 64-bit integers and the other uses 32-bit integers). The computer language benchmarks game was initially conceived to determine a ballpark for how slow interpreted and managed languages are compared to C. Quantifying the overhead of interpreters and runtimes is its raison d'etre, and it shows. When it comes to comparing low-level systems languages that have no runtime to speak of, the best it can do is attempt to quantify the quality of each backend's code generator (it's a missed opportunity that it doesn't include Clang for comparison with GCC). (And yes, I understand that the benchmarks game contains repeated massive disclaimers that people should not take the performance results as a means of serious comparison. Internet commentators remain undeterred.) If you're just trying to argue that the methodology used in the OP is poor, then obviously we're in agreement (was there ever any doubt?).
- igouy 10y ago
  > The computer language benchmarks game was initially conceived to determine a ballpark for how slow interpreted and managed languages are compared to C. Today the benchmarks game is referred to from the Rust FAQ. What is one to do? https://www.rust-lang.org/en-US/faq.html#performance https://www.rust-lang.org/en-US/faq.html#performance Back in the previous millenia -- "[Doug Bagley's] goal was to compare all the major scripting languages. Then [Doug Bagley] started adding in some compiled languages for comparison…" http://web.archive.org/web/20010125021400/http://www.bagley.org/~doug/shootout/#News http://web.archive.org/web/20010125021400/http://www.bagley....
- igouy 10y ago
  > …repeated massive disclaimers that people should not take the performance results as a means of serious comparison. Actually one small disclaimer -- "Non-motivation: We are profoundly uninterested in claims that these measurements, of a few tiny programs, somehow define the relative performance of programming languages." There's much more about PL benchmarks in general.
deleted 10y ago
[deleted]

by acqq 10y ago

kibwen 10y ago

> The programs on the site fit the rules defined on it, and the rules include the verification of the results I'm not sure what you're refuting? Verifying that the results match is a necessary but insufficient quality for ensuring comparability. If the algorithm could be the same in each language but isn't (e.g. quicksort vs. bogosort), then that's not a valid comparison if your objective is to determine the overhead imposed by the language implementation itself (and if you're not trying to determine language implementation overhead, then what are you measuring?). Likewise if the implementation details could be the same in each language but aren't (e.g. if one uses 64-bit integers and the other uses 32-bit integers). The computer language benchmarks game was initially conceived to determine a ballpark for how slow interpreted and managed languages are compared to C. Quantifying the overhead of interpreters and runtimes is its raison d'etre, and it shows. When it comes to comparing low-level systems languages that have no runtime to speak of, the best it can do is attempt to quantify the quality of each backend's code generator (it's a missed opportunity that it doesn't include Clang for comparison with GCC). (And yes, I understand that the benchmarks game contains repeated massive disclaimers that people should not take the performance results as a means of serious comparison. Internet commentators remain undeterred.) If you're just trying to argue that the methodology used in the OP is poor, then obviously we're in agreement (was there ever any doubt?).

igouy 10y ago
> The computer language benchmarks game was initially conceived to determine a ballpark for how slow interpreted and managed languages are compared to C. Today the benchmarks game is referred to from the Rust FAQ. What is one to do? https://www.rust-lang.org/en-US/faq.html#performance https://www.rust-lang.org/en-US/faq.html#performance Back in the previous millenia -- "[Doug Bagley's] goal was to compare all the major scripting languages. Then [Doug Bagley] started adding in some compiled languages for comparison…" http://web.archive.org/web/20010125021400/http://www.bagley.org/~doug/shootout/#News http://web.archive.org/web/20010125021400/http://www.bagley....
igouy 10y ago
> …repeated massive disclaimers that people should not take the performance results as a means of serious comparison. Actually one small disclaimer -- "Non-motivation: We are profoundly uninterested in claims that these measurements, of a few tiny programs, somehow define the relative performance of programming languages." There's much more about PL benchmarks in general.

deleted 10y ago

[deleted]