3 ms·

Yes, I understand what VM implementors can derive from individual results of each and every micro-benchmark. [I usually go as far as saying that only VM implem

by mraleph 13y ago

Yes, I understand what VM implementors can derive from individual results of each and every micro-benchmark. [I usually go as far as saying that only VM implementors and people with deep enough understanding of VMs should ever micro-benchmark and pay attention to micro-benchmarking results] I would however argue that you could just run each microbenchmark separately and report results without even mentioning that those benchmarks (in their short running forms) constitute SunSpider. Another thing that you could have done is either disable transcendental cache for IonMonkey generated code altogether (or add transcendental cache in Java / C++ code) and reported pure JavaScript results on the main, prominently visible graph. > asm.js was generally able to hold its own I am sure that you wanted to say OdinMonkey here instead of asm.js. asm.js is a set of syntactic and typing rules, it does not have any performance per se. OdinMonkey is an implementation. I have seen people conflate these two things together again and again.

kannanvijayan 13y ago
I disagree that only VM implementors should think about these things. It suggests a degree of overspecialization that I think is good to avoid. Even if one is not a VM implementor, it's always good to have a good grasp on critical evaluation of results. That's something I tried to promote with my article: to encourage people to think more deeply about benchmarks than simply noting the final number and proceeding. I considered disabling the math cache in SpiderMonkey and using those scores, but it seemed inappropriate. I don't know to what extent the tuning and other optimizations are otherwise targeted to SunSpider. The math cache is obvious, but there may be many non-obvious tunings. Let's face it, JS engine devs have been trying to improve scores for a long time now. To what degree have tuning of GC, of when-to-jit, of what-to-jit, of when-to-inline, of what-to-inline, of the numerous thresholds.. been targeted at SunSpider? I didn't feel comfortable just removing the math cache and simply stating "well now we've leveled the playing field so native JS is not at an advantage". Also, you're entirely correct about OdinMonkey vs asm.js. It's too easy to use "asm.js" as a shorthand for particular implementations. It's something people tend to do (e.g. "java" to mean "the Sun JVM"), but I should definitely try to avoid that.

by mraleph 13y ago

kannanvijayan 13y ago

I disagree that only VM implementors should think about these things. It suggests a degree of overspecialization that I think is good to avoid. Even if one is not a VM implementor, it's always good to have a good grasp on critical evaluation of results. That's something I tried to promote with my article: to encourage people to think more deeply about benchmarks than simply noting the final number and proceeding. I considered disabling the math cache in SpiderMonkey and using those scores, but it seemed inappropriate. I don't know to what extent the tuning and other optimizations are otherwise targeted to SunSpider. The math cache is obvious, but there may be many non-obvious tunings. Let's face it, JS engine devs have been trying to improve scores for a long time now. To what degree have tuning of GC, of when-to-jit, of what-to-jit, of when-to-inline, of what-to-inline, of the numerous thresholds.. been targeted at SunSpider? I didn't feel comfortable just removing the math cache and simply stating "well now we've leveled the playing field so native JS is not at an advantage". Also, you're entirely correct about OdinMonkey vs asm.js. It's too easy to use "asm.js" as a shorthand for particular implementations. It's something people tend to do (e.g. "java" to mean "the Sun JVM"), but I should definitely try to avoid that.