Exhaustive statistical analysis and over-generalization based on inappropriate data - garbage-in garbage-out. --- >> "… the average Go program is 18.7 times s

Exhaustive statistical analysis and over-generalization based on inappropriate data - garbage-in garbage-out. --- >> "… the average Go program is 18.7 times slower than the average C program. Programs in other languages are much slower than Go programs … 4.6–13.7 times slower than Go on average" page 7 << Elephant in the room -- What kind-of programs did the Rosetta Code program authors intend to write? Did they write their programs as-if run-time performance did not matter? Did they trade-off program performance for pretty-code / shorter programs? Did they trade-off program sophistication for program clarity? For example, look at the gap between the Shortest C++ programs on the benchmarks game and the fastest. (I pestered a C++ program contributor to write shorter programs.) http://benchmarksgame.alioth.debian.org/u64/code-used-time-used-shapes.php#shortest http://benchmarksgame.alioth.debian.org/u64/code-used-time-u... >> " …every large project is likely to include a small set of core functionalities whose quality, performance, and reliability significantly influences the whole system’s; Rosetta Code programs are indicative of such core functionalities." page 10 << Large projects are likely to identify that small set of core functionalities and the programmers are likely to make different trade-offs for those parts of the project, for example -- optimize performance over short code, optimize performance over programming time. --- >> To have reliable performance measurements, the scripts repeat each execution 6 times; the timing of the first execution is discarded (to fairly accommodate bytecode languages that load virtual machines from disk: it is only in the first execution that the virtual machine is loaded from disk, with corresponding possibly significant one-time overhead; in the successive executions the virtual machine is read from cache, with only limited overhead). page 4 << The method does not take into account JVM dynamic optimization, and because we are not told the actual run-times of the programs there is no way to even guess whether the program has run long enough to gain the benefits. --- >> "Which programming languages are less failure prone? To answer this question, we measure runtime failures of solutions of tasks T EXEC marked for execution that compile without errors or timeout. … Thus, a terminating program fails if it returns an exit code other than 0." page 8 << Why would program failure be a property of a programming language rather than an indication that the programmer did not test their program? (Or an indication that the program was written for some other implementation of the language? Or written for a workload smaller than the workload tested, given available memory?) Even if the program did not fail, is the result correct? --- >> "… we maintain that paradigms still significantly influence the typical “style” in which programs are written, and it is natural to associate major programming languages to a specific style based on their Rosetta Code programs. For example, even if Python offers classes and other object-oriented features, practically no solutions in Rosetta Code use them." page 11 << Not whether the Python programs should be considered imperative because they did not use OO features, but whether the Java and C# programs should not be considered OO because they did not go beyond the boiler-plate required by the language.