30 ms·
> 1) source code size is compared This is new to me (the Benchmarks Game has been around for well over a decade). > 2) correctness is checked This is par for
by jcranmer 6y ago
> 1) source code size is compared
This is new to me (the Benchmarks Game has been around for well over a decade).
> 2) correctness is checked
This is par for the course of benchmarks. A benchmark that is incorrectly compiled provides no useful information whatsoever about performance. It's not really what people mean when they complain about correctness issues in the compiler (which is generally referred to as soundness in literature).
> 3) time "to complete and log all make actions" (build the program) is also logged
Performance in a compiler is usually taken to be a combination of runtime, code size, and compile time, although all three components are not weighted equally.
- igouy 6y agoBack on May 23 2006, a sortable column of GZip source code size measurements: https://web.archive.org/web/20060523195137/http://shootout.alioth.debian.org/gp4/benchmark.php?test=spectralnorm&lang=all https://web.archive.org/web/20060523195137/http://shootout.a... https://web.archive.org/web/20060519213743/http://shootout.alioth.debian.org/gp4/faq.php#codelines https://web.archive.org/web/20060519213743/http://shootout.a... > Performance in a compiler is usually taken to be a combination of runtime, code size, and compile time The benchmarks game shows source code size, not compiled code size.