4 ms·

> …"Benchmarks Game" … compares languages solely on the basis of performance. Not true. 1) source code size is compared https://benchmarksgame-team.pages.deb

by igouy 6y ago

> …"Benchmarks Game" … compares languages solely on the basis of performance. Not true. 1) source code size is compared https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/spectralnorm-gz.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... https://benchmarksgame-team.pages.debian.net/benchmarksgame/how-programs-are-measured.html#source-code https://benchmarksgame-team.pages.debian.net/benchmarksgame/... 2) correctness is checked https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/nbody-fsharpcore-1.html#log https://benchmarksgame-team.pages.debian.net/benchmarksgame/... 3) time "to complete and log all make actions" (build the program) is also logged

jcranmer 6y ago
> 1) source code size is compared This is new to me (the Benchmarks Game has been around for well over a decade). > 2) correctness is checked This is par for the course of benchmarks. A benchmark that is incorrectly compiled provides no useful information whatsoever about performance. It's not really what people mean when they complain about correctness issues in the compiler (which is generally referred to as soundness in literature). > 3) time "to complete and log all make actions" (build the program) is also logged Performance in a compiler is usually taken to be a combination of runtime, code size, and compile time, although all three components are not weighted equally.
- igouy 6y ago
  Back on May 23 2006, a sortable column of GZip source code size measurements: https://web.archive.org/web/20060523195137/http://shootout.alioth.debian.org/gp4/benchmark.php?test=spectralnorm&lang=all https://web.archive.org/web/20060523195137/http://shootout.a... https://web.archive.org/web/20060519213743/http://shootout.alioth.debian.org/gp4/faq.php#codelines https://web.archive.org/web/20060519213743/http://shootout.a... > Performance in a compiler is usually taken to be a combination of runtime, code size, and compile time The benchmarks game shows source code size, not compiled code size.

by igouy 6y ago

jcranmer 6y ago

> 1) source code size is compared This is new to me (the Benchmarks Game has been around for well over a decade). > 2) correctness is checked This is par for the course of benchmarks. A benchmark that is incorrectly compiled provides no useful information whatsoever about performance. It's not really what people mean when they complain about correctness issues in the compiler (which is generally referred to as soundness in literature). > 3) time "to complete and log all make actions" (build the program) is also logged Performance in a compiler is usually taken to be a combination of runtime, code size, and compile time, although all three components are not weighted equally.

igouy 6y ago
Back on May 23 2006, a sortable column of GZip source code size measurements: https://web.archive.org/web/20060523195137/http://shootout.alioth.debian.org/gp4/benchmark.php?test=spectralnorm&lang=all https://web.archive.org/web/20060523195137/http://shootout.a... https://web.archive.org/web/20060519213743/http://shootout.alioth.debian.org/gp4/faq.php#codelines https://web.archive.org/web/20060519213743/http://shootout.a... > Performance in a compiler is usually taken to be a combination of runtime, code size, and compile time The benchmarks game shows source code size, not compiled code size.