Cheating on SPEC benchmarks? Surely that has never happened before... "We conclude this paper by citing cases [of] compiler optimizations specifically designed

Cheating on SPEC benchmarks? Surely that has never happened before... "We conclude this paper by citing cases [of] compiler optimizations specifically designed for SPEC programs, in which performance decreases drastically or the computed results are incorrect if the compiled program does not exactly match the SPEC benchmark program. These findings show that the fairness and value of the popular SPEC benchmarks are questionable."[1] "Issue: Special library for a specific benchmark... Issue: Pattern matching for the inner loop of a specific benchmark... Issue: Compilers optimize too much for the SPEC CPU benchmarks... Issue: The compiler optimization parameters (flags) used are unrealistic..."[2] See also: Goodhart's Law[3]. [1] N. Mirghafori, M. Jacoby, D. Patterson, "Truth in SPEC benchmarks", ACM SIGARCH Computer Architecture News, 1995 [2] R. Weicker, "SPEC OSG Benchmarks: History, Use and Current Issues", in "Performance Evaluation and Benchmarking with Realistic Applications", R. Eigenmann (Ed.), 2001 [3] https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law