Disclaimer: I'm a JS JIT developer. The use of Sunspider performance to talk about JS application performance is unfortunate, and undermines a significant part

Disclaimer: I'm a JS JIT developer. The use of Sunspider performance to talk about JS application performance is unfortunate, and undermines a significant part of the argument made in the article. Sunspider is a great benchmark to use to talk about web _page_ performance. This is how JITs work: 1. Javascript code starts executing in a slow environment. 2. In this execution mode, a lot of overhead is spent collecting type information so that the hot code can be optimized. 3. After some period of time, the hot code is compiled with an optimizing compiler, which is a very expensive task. 4. The compiled code starts running. For example, in Mozilla's JS engine, these are the actual numbers associated with optimization: 1. A function starts off running in the interpreter 2. After it gets called 10 times (or a loop within it goes around 10 times), it gets compiled with the Baseline JIT, which performs heavyweight bytecode analysis on the javascript code, generates unoptimized machine code which is instrumented to acquire information about the executing JS. 3. After 1000 iterations (most of that running within Baseline jitcode), the function gets compiled with our optimizing compiler, IonMonkey. How does this relate to Sunspider? Well, if you actually run Sunspider, you'll notice that the entire benchmark takes about 0.2 seconds or less on a modern machine (about 1.5 seconds on a modern phone). It's an extremely tiny benchmark (in terms of computation heaviness). Comparatively, the Octane suite, while containing fewer benchmarks, actually takes 100 times or so longer to execute than sunspider. How many apps do you use that run for less than 1.5 seconds? This is the problem with using Sunspider to talk about app performance. Sunspider doesn't measure JIT performance very well at all. When JIT engines execute Sunspider, they spend a relatively large amount of that time just collecting the necessary information before they can start optimizing. Then, soon after we spend all that effort optimizing the code, the benchmark ends, without really taking advantage of all that fast code. Very often, JIT optimizations that speed up computationally heavy benchmarks actually regress Sunspider. This is because the optimizations add some extra weight to the initial analysis that happens before optimized code is generated. This hurts sunspider because the initial analysis section is much more heavily weighted than later optimized performance. Sunspider's runtime characteristics make it a good benchmark for measuring web PAGE performance. Web pages are much more likely to contain code that executes for short periods of time (a few ms), and then stops. But apps have much different runtime characteristics. Apps don't execute code for a few milliseconds and then stop. They run for dozens of seconds, minimally, and often minutes at a time. A JIT engine has much more opportunity to generate good optimized code, and have that optimized code taken advantage of over more of the execution time. It's unfortunate that such a fundamental misunderstanding of the characteristics and nature of optimized javascript is used to drive an argument about it.