4 ms·

It feels to me like current benchmarks all measure execution time, not user time. Let's say you go from a reasonably sized web page to another, and then hit the

by cwyers 11y ago

It feels to me like current benchmarks all measure execution time, not user time. Let's say you go from a reasonably sized web page to another, and then hit the back button. How long to load the previous page? Right click on a link and select open in new tab. How long does it take to open a new tab? It's definitely part of the user's perception of the browser's speed, but I don't think stuff like SunSpider and Octane even tries to measure it.

ripberge 11y ago
Good point. I have a surface and want to use IE 11 more because its touch screen friendly, however the back button performance is slow sometimes. Can be very frustrating, so I typically use Chrome. Hopefully Edge fixes that.
pcwalton 11y ago
Heavily related to this, one key insight that I think has been neglected is that JavaScript benchmarks do a bad job of measuring cold code. Most code on the Web is cold, because much Web browsing consists of fresh page loads. Heavily optimizing JITs are exactly what you don't want for cold code: time spent compiling is time you could have been spending running the code. But benchmarks consist of running the same code repeatedly in a long-running loop, which encourages development of sophisticated, highly-optimizing JITs that spend a lot of time in up-front compilation, which end up being totally pointless for JS whose sole purpose is to run once to place an ad somewhere on the page. It's easy to forget that JavaScript is still primarily a scripting language used to perform high-level application tasks. It's awesome that JS has gone so far low-level, and we should continue to push it as far as it can, but I think the bread and butter that users care about is not so much heavy number crunching but rather quick cold code that needs to run once or twice and get out of the way.
- jgraham 11y ago
  Related: http://research.microsoft.com/apps/pubs/default.aspx?id=118663 http://research.microsoft.com/apps/pubs/default.aspx?id=1186... I don't know if it's a coincidence that around the time that paper was released Microsoft were considerably behind the curve in terms of js engine performance. The paper also misses the fact that excellent hot-code performance opens up new frontiers for the web like games, photo editing, and similar compute-intensive applications that are not well represented by their sample of popular sites since those are, by definition, limited by what was performant in extant browsers.
- jcranmer 11y ago
  Well, yes and no. Pure JavaScript benchmarks (SunSpider, Octane, Kraken) are all more or less microbenchmarks, at least to the degree that they're testing only one specific kernel of an application (even if Octane and Kraken are justifable since they're pulling largish kernels from real code). A pure JS "cold" benchmark would be an even poorer microbenchmark, since it carries a large presumption that the cold JS is the bottleneck. In real JS code of the stuff you're talking about, you might be able to easily get measurable performance gains by, say, improving FFI between JS and DOM code, or improving the DOM code itself. A better benchmark would be measuring page load times (or energy usage) or dynamic operations that included such cold JS code, since it would illustrate where performance improvements would be most beneficial.
- arsa 11y ago
  Have you actually looked at some of these benchmarks? For example, Octane contains "Code loading", sunspider has "code decompression" (not true code loading, but at least it does de-minification). Of course it's important, and all borwsers are developed with this in mind - but it's not possible to create reproducable test suite (with a score given) that will test actual sites - sites change
- haberman 11y ago
  > Heavily optimizing JITs are exactly what you don't want for cold code: time spent compiling is time you could have been spending running the code. I agree with your overall point of cold cold performance being paramount. But I don't think this follows. A heavily optimizing JIT can have very good cold-code performance if the JIT is especially fast and/or it makes especially good decisions about when to JIT. The important trait is short break-even time: how long after JIT-ting some code that you break even on the investment of JITting time. When all I knew was the JVM I assumed that JIT-compiling inherently had a long warm-up / break-even time. Now that I've experienced LuaJIT I know this isn't true. I haven't benchmarked the major JIT-ting JS engines but they seem pretty good at this too?
  - pcwalton 11y ago
    By "a heavily optimizing JIT" I mean "the specific mode of a larger JIT-enabled engine that compiles slowly but emits good code". So IonMonkey, Crankshaft, SpeculativeJIT/FTL are all JITs that live inside a larger engine that switches between them as needed. In JS land we usually call the individual modes "JITs" even though they may (or may not) share a bunch of code.
- sanxiyn 11y ago
  JSBench is a good benchmark measuring cold performance. I recommend everybody try running it. It is known that Safari wins this benchmark, by a large margin.
  - randomfool 11y ago
    Unfortunately the way it measures JS perf is questionable- it essentially records all operations during page load then replays them for the benchmark. This has the effect of unrolling all of the loops and disabling many of the optimizations that would have been applied to the original code.
- pjmlp 11y ago
  > I think the bread and butter that users care about is not so much heavy number crunching but rather quick cold code that needs to run once or twice and get out of the way. I care about HTML and CSS with as little JavaScript as possible. The best code to optimize is the one isn't there in first place.

by cwyers 11y ago

ripberge 11y ago

Good point. I have a surface and want to use IE 11 more because its touch screen friendly, however the back button performance is slow sometimes. Can be very frustrating, so I typically use Chrome. Hopefully Edge fixes that.

pcwalton 11y ago

Heavily related to this, one key insight that I think has been neglected is that JavaScript benchmarks do a bad job of measuring cold code. Most code on the Web is cold, because much Web browsing consists of fresh page loads. Heavily optimizing JITs are exactly what you don't want for cold code: time spent compiling is time you could have been spending running the code. But benchmarks consist of running the same code repeatedly in a long-running loop, which encourages development of sophisticated, highly-optimizing JITs that spend a lot of time in up-front compilation, which end up being totally pointless for JS whose sole purpose is to run once to place an ad somewhere on the page. It's easy to forget that JavaScript is still primarily a scripting language used to perform high-level application tasks. It's awesome that JS has gone so far low-level, and we should continue to push it as far as it can, but I think the bread and butter that users care about is not so much heavy number crunching but rather quick cold code that needs to run once or twice and get out of the way.

jgraham 11y ago
Related: http://research.microsoft.com/apps/pubs/default.aspx?id=118663 http://research.microsoft.com/apps/pubs/default.aspx?id=1186... I don't know if it's a coincidence that around the time that paper was released Microsoft were considerably behind the curve in terms of js engine performance. The paper also misses the fact that excellent hot-code performance opens up new frontiers for the web like games, photo editing, and similar compute-intensive applications that are not well represented by their sample of popular sites since those are, by definition, limited by what was performant in extant browsers.
jcranmer 11y ago
Well, yes and no. Pure JavaScript benchmarks (SunSpider, Octane, Kraken) are all more or less microbenchmarks, at least to the degree that they're testing only one specific kernel of an application (even if Octane and Kraken are justifable since they're pulling largish kernels from real code). A pure JS "cold" benchmark would be an even poorer microbenchmark, since it carries a large presumption that the cold JS is the bottleneck. In real JS code of the stuff you're talking about, you might be able to easily get measurable performance gains by, say, improving FFI between JS and DOM code, or improving the DOM code itself. A better benchmark would be measuring page load times (or energy usage) or dynamic operations that included such cold JS code, since it would illustrate where performance improvements would be most beneficial.
arsa 11y ago
Have you actually looked at some of these benchmarks? For example, Octane contains "Code loading", sunspider has "code decompression" (not true code loading, but at least it does de-minification). Of course it's important, and all borwsers are developed with this in mind - but it's not possible to create reproducable test suite (with a score given) that will test actual sites - sites change
haberman 11y ago
> Heavily optimizing JITs are exactly what you don't want for cold code: time spent compiling is time you could have been spending running the code. I agree with your overall point of cold cold performance being paramount. But I don't think this follows. A heavily optimizing JIT can have very good cold-code performance if the JIT is especially fast and/or it makes especially good decisions about when to JIT. The important trait is short break-even time: how long after JIT-ting some code that you break even on the investment of JITting time. When all I knew was the JVM I assumed that JIT-compiling inherently had a long warm-up / break-even time. Now that I've experienced LuaJIT I know this isn't true. I haven't benchmarked the major JIT-ting JS engines but they seem pretty good at this too?
- pcwalton 11y ago
  By "a heavily optimizing JIT" I mean "the specific mode of a larger JIT-enabled engine that compiles slowly but emits good code". So IonMonkey, Crankshaft, SpeculativeJIT/FTL are all JITs that live inside a larger engine that switches between them as needed. In JS land we usually call the individual modes "JITs" even though they may (or may not) share a bunch of code.
sanxiyn 11y ago
JSBench is a good benchmark measuring cold performance. I recommend everybody try running it. It is known that Safari wins this benchmark, by a large margin.
- randomfool 11y ago
  Unfortunately the way it measures JS perf is questionable- it essentially records all operations during page load then replays them for the benchmark. This has the effect of unrolling all of the loops and disabling many of the optimizations that would have been applied to the original code.
pjmlp 11y ago
> I think the bread and butter that users care about is not so much heavy number crunching but rather quick cold code that needs to run once or twice and get out of the way. I care about HTML and CSS with as little JavaScript as possible. The best code to optimize is the one isn't there in first place.