3 ms·

Every time I see JS performance data like this, I'm always mystified to see lots of error ranges of a couple percent. Working on a JS game has made me dive pret

by fenomas 10y ago

Every time I see JS performance data like this, I'm always mystified to see lots of error ranges of a couple percent. Working on a JS game has made me dive pretty deeply into JS performance (on Chrome at least), and I constantly find that the hardest part of performance tuning is getting any kind of consistent results to tune. For a random example, I tried reproducing the "Swap rows" test for Vanilla.js. The author found this took 7.83 +/- 0.18 ms, with five warmup iterations. On my machine, running the same command six times consistently gives results like these: swapRows took 10.944999999999709 swapRows took 1.3250000000007276 swapRows took 1.3200000000006185 swapRows took 1.269999999999527 swapRows took 11.875 swapRows took 1.2750000000005457 Also, and orthogonal to the above, I've always found that even when performance is consistent across repeated trials, it varies widely between page reloads. (This isn't at all surprising, since v8's optimizing compiler isn't necessarily going to make the same choices every time - but it still makes things very hard to tune). As an example of this, I tried repeating the "append to large table" test. The author's results were 225.3 +/- 1.89ms. On my (similar) machine, trying this five times (reloading the page between) varied between 225 and 329ms. Am I missing something here? I've looked pretty hard into v8 perf tuning and various benchmarking libraries, and I've just never seen the kind of consistent results one always finds in comparisons like these.

beefsack 10y ago
It's hard to get consistent performance when you have any form of memory management, caching, or a JIT behind the scenes. I feel like this is just something you have to accept when writing code in a high level language.
- fenomas 10y ago
  Sure, and in tuning my game I've tried to work out ways of accounting for this. My point is basically that JS performance data with really small error ranges makes me suspicious about whether the data is meaningful or not.
krausest 10y ago
You're basically right. Thus we've decided to perform 10 runs for each benchmark and drop the worst 4. This eliminates cases where the gc or a background system process causes a slow run. In consequence this also strongly reduces variance, but is blind on the gc eye. It would be interesting to include gc, but I have no idea how to do it in a real good way. The results are comparable (but nowhere equal) between runs and the difference between the frameworks is in most cases large enough that the ranking stays consistent (but you wouldn't prefer preact to cycle.js v7 for having a 0.01 better result in the average slow down, would you? For ember, mithril and cycle.js v6 performance might be something to consider depending on your use case.). If you look at the console output this is just an approximation, the real measurement is performed on data from chrome's timeline using selenium. The console measurement uses a timer to guess when rendering was performed. Of course this is more inaccurate than using the timeline.
- isoos 10y ago
  I think an equally interesting result would be to keep only the worst 1-4 benchmark results and compare those. In my experience some frameworks put much more long-term memory pressure on the GC, and by discarding those, your benchmark turns a blind eye on that. In another words, I'd be interested more on the worst possible performance of a given framework, than their best (assuming the app is written in a reasonably good way otherwise). Also, +1 to the sibling post that asks for longer runs, that would solve the GC question by including it altogether!
- yabb 10y ago
  Do the benchmarks take a long time? Why not as many runs as necessary to give you a tight confidence interval?
  - krausest 10y ago
    Running all the benchmarks takes a few hours. I'll keep that in mind for the next round. Let's see if that works nicely. Would you still drop outliers?
- fenomas 10y ago

by fenomas 10y ago

beefsack 10y ago

It's hard to get consistent performance when you have any form of memory management, caching, or a JIT behind the scenes. I feel like this is just something you have to accept when writing code in a high level language.

fenomas 10y ago
Sure, and in tuning my game I've tried to work out ways of accounting for this. My point is basically that JS performance data with really small error ranges makes me suspicious about whether the data is meaningful or not.

krausest 10y ago

You're basically right. Thus we've decided to perform 10 runs for each benchmark and drop the worst 4. This eliminates cases where the gc or a background system process causes a slow run. In consequence this also strongly reduces variance, but is blind on the gc eye. It would be interesting to include gc, but I have no idea how to do it in a real good way. The results are comparable (but nowhere equal) between runs and the difference between the frameworks is in most cases large enough that the ranking stays consistent (but you wouldn't prefer preact to cycle.js v7 for having a 0.01 better result in the average slow down, would you? For ember, mithril and cycle.js v6 performance might be something to consider depending on your use case.). If you look at the console output this is just an approximation, the real measurement is performed on data from chrome's timeline using selenium. The console measurement uses a timer to guess when rendering was performed. Of course this is more inaccurate than using the timeline.

isoos 10y ago
I think an equally interesting result would be to keep only the worst 1-4 benchmark results and compare those. In my experience some frameworks put much more long-term memory pressure on the GC, and by discarding those, your benchmark turns a blind eye on that. In another words, I'd be interested more on the worst possible performance of a given framework, than their best (assuming the app is written in a reasonably good way otherwise). Also, +1 to the sibling post that asks for longer runs, that would solve the GC question by including it altogether!
yabb 10y ago
Do the benchmarks take a long time? Why not as many runs as necessary to give you a tight confidence interval?
- krausest 10y ago
  Running all the benchmarks takes a few hours. I'll keep that in mind for the next round. Let's see if that works nicely. Would you still drop outliers?
fenomas 10y ago