We did not want to “adapt” the algorithm to each language to improve speed, because it will open the door to too many “uncontrolled variables” (how good a progr

We did not want to “adapt” the algorithm to each language to improve speed, because it will open the door to too many “uncontrolled variables” (how good a programmer in that language you are and so on). The only partial exception was with Mathematica for reasons explained in the paper. This is rather half-baked reasoning. The only ways I can think of to control for programmer skill or at least account for it is to 1. Use "idiomatic" code for all languages. 2. Use "optimized" code for all languages. 3. Use "real" code for all languages with a large and detailed sample set. In number 3 you aren't "controlling" the variables so much as diligently including them in the observations. The baseline skill of a programmer sounds to me more like an unknown than a variable-- something more worth studying first than controlling.