5 ms·

Speaking from the perspective of a (former) V8 engineer, SunSpider is crappy benchmark, and I can't believe people still track it. It has misdirected JavaScript

by titzer 3y ago

Speaking from the perspective of a (former) V8 engineer, SunSpider is crappy benchmark, and I can't believe people still track it. It has misdirected JavaScript performance engineering work for going on 15 years now, leading to all kinds of weird contortions in JavaScript engines that do not benefit real world sites. Years ago, V8 switched away from microbenchmarks for the most part. See: https://benediktmeurer.de/2016/12/16/the-truth-about-traditional-javascript-benchmarks/ https://benediktmeurer.de/2016/12/16/the-truth-about-traditi... edit to add: Don't take the above to mean that Firefox's performance improvements aren't significant, and welcome. It's just that SunSpider is the absolute worst benchmark to look at.

Waterluvian 3y ago
This isn’t an agreement or disagreement: Having benchmarks based on real-world sites seems like both a good and a bad idea. Good for obvious reasons. Bad because it creates a feedback loop. People develop based on performance. Performance is improved based on what people develop. Did you/your colleagues tackle this in some way?
- titzer 3y ago
  Here's one example from 2016: https://v8.dev/blog/real-world-performance https://v8.dev/blog/real-world-performance
olliej 3y ago
In fairness, when sun spider was first released it did accurately reflect the core pain points in JS engines, as present on the web. That applied to V8 as well (though V8 introduced its own benchmark suite as well on release, it was overly focused on property access, and insufficiently on the parts of initial page load that sun spider just happened to cover). Of course all of that was _15 years ago_, no benchmark from back then is really reflective of anything useful, and they are largely no longer valuable for anything other than catching regressions.
mhh__ 3y ago
As the saying goes any compiler engineer can make their compiler the fastest if they know what the code is.
tgv 3y ago
I agree, but speaking as someone who's been using JS for computationally medium-intensive tasks: Firefox and Chrome don't differ that much. For most users, the difference is negligible.
hiimkeks 3y ago
The link you posted promotes Speedometer, where Firefox reportedly already surpassed Chrome. https://news.ycombinator.com/item?id=36770883 https://news.ycombinator.com/item?id=36770883
MegaDeKay 3y ago
Firefox tracks their performance across many different benchmarks. Here is the performance vs Chrome over the past year on many different ones. It is encouraging to see steady improvements being made on a substantial number of them over time. https://arewefastyet.com/win10/benchmarks/overview?numDays=365 https://arewefastyet.com/win10/benchmarks/overview?numDays=3...
- yborg 3y ago
  It appears that Chrome still wins most of these. Even the touted Speedometer win has apparently been trumped by Chrome over the last couple of weeks. The competition doesn't stand still. Which is good, imagine if Firefox threw in the towel, Chrome would become IE 6 in short order.
  - nwienert 3y ago
    Safari is faster than either across almost all dimensions, lighter on battery, nicer UI, better standards support generally than FF and even matches chrome for what I care about. It’s the clear best browser imo.
    - pmontra 3y ago
      I can run Chrome and Firefox on my Debian laptop and on my Android devices. How do I benchmark Safari there? I can't, so Safari is barely relevant to me. Nice to know that it's faster than the other two browsers on Macs, but I'm not very surprised. Apple should be able to integrate it with their hardware and OS more than Google and Mozilla.
    - isaacremuant 3y ago
      Customization and freedom means Firefox is always the best for me. How's the safari world of add-on's? Do you have tab containers? What about mobile? Can you block ads?
    - NicoJuicy 3y ago
      On Mac? Probably because only Safari has access to the private api's
IainIreland 3y ago
To be clear: we track it in the sense that we still have it running as part of old dashboards. It's cheap, and every so often it catches an unintended regression. The SpiderMonkey team is very much on board with ignoring microbenchmarks. We care about Speedometer, because it correlates reasonably well with real-world performance. We pick and choose which parts of Jetstream2 we bother with. Anything older than that is strictly a measuring stick, rather than a target: it's nice when the numbers go up, but it will only happen as a side-effect of work we're doing for other reasons. Notably, all the SunSpider wins on this graph are side-effects of Speedometer work we've been doing this year.
- titzer 3y ago
  I understand the logic, but I never understood why we (and I mean, collectively, the JS VM engineers across the browser vendors) didn't just rewrite the benchmark (or fix it) and name it SunSpider 2. I suggested this many times. It's not that much code.
  - londons_explore 3y ago
    Presumably because replicating the mix of operations used in real react/jquery code all over the web accurately isn't trivial...
  - IainIreland 3y ago
    Yeah, I've been making this argument for years too. Speedometer 3 is in the works, and it's moving in this direction. On multiple occasions we've done performance analysis of the latest draft version, noticed a particular chunk of code is hot, considered what it would take to optimize it, and then changed it because we don't think it reflects real-world usage. EDIT: And I guess the answer for "why not fix SunSpider specifically?" is that once you've done the work to turn it into something worth optimizing, it no longer bears any resemblance to the original. It probably looks a lot more like Speedometer, honestly.
    - titzer 3y ago
      > EDIT: And I guess the answer for "why not fix SunSpider specifically?" is that once you've done the work to turn it into something worth optimizing, it no longer bears any resemblance to the original. It probably looks a lot more like Speedometer, honestly. I get that, yeah, but superseding the old name is probably worth it though; the internet is full of people who never get the memo except via version numbers :) I tongue-in-cheek proposed that we match the Sunspider ASTs in V8 and either make it infinitely fast or infinitely slow. Now that will kill a benchmark once and for all!

by titzer 3y ago

Waterluvian 3y ago

This isn’t an agreement or disagreement: Having benchmarks based on real-world sites seems like both a good and a bad idea. Good for obvious reasons. Bad because it creates a feedback loop. People develop based on performance. Performance is improved based on what people develop. Did you/your colleagues tackle this in some way?

titzer 3y ago
Here's one example from 2016: https://v8.dev/blog/real-world-performance https://v8.dev/blog/real-world-performance

olliej 3y ago

In fairness, when sun spider was first released it did accurately reflect the core pain points in JS engines, as present on the web. That applied to V8 as well (though V8 introduced its own benchmark suite as well on release, it was overly focused on property access, and insufficiently on the parts of initial page load that sun spider just happened to cover). Of course all of that was _15 years ago_, no benchmark from back then is really reflective of anything useful, and they are largely no longer valuable for anything other than catching regressions.

mhh__ 3y ago

As the saying goes any compiler engineer can make their compiler the fastest if they know what the code is.

tgv 3y ago

I agree, but speaking as someone who's been using JS for computationally medium-intensive tasks: Firefox and Chrome don't differ that much. For most users, the difference is negligible.

hiimkeks 3y ago

The link you posted promotes Speedometer, where Firefox reportedly already surpassed Chrome. https://news.ycombinator.com/item?id=36770883 https://news.ycombinator.com/item?id=36770883

MegaDeKay 3y ago

Firefox tracks their performance across many different benchmarks. Here is the performance vs Chrome over the past year on many different ones. It is encouraging to see steady improvements being made on a substantial number of them over time. https://arewefastyet.com/win10/benchmarks/overview?numDays=365 https://arewefastyet.com/win10/benchmarks/overview?numDays=3...

yborg 3y ago
It appears that Chrome still wins most of these. Even the touted Speedometer win has apparently been trumped by Chrome over the last couple of weeks. The competition doesn't stand still. Which is good, imagine if Firefox threw in the towel, Chrome would become IE 6 in short order.
- nwienert 3y ago
  Safari is faster than either across almost all dimensions, lighter on battery, nicer UI, better standards support generally than FF and even matches chrome for what I care about. It’s the clear best browser imo.
  - pmontra 3y ago
    I can run Chrome and Firefox on my Debian laptop and on my Android devices. How do I benchmark Safari there? I can't, so Safari is barely relevant to me. Nice to know that it's faster than the other two browsers on Macs, but I'm not very surprised. Apple should be able to integrate it with their hardware and OS more than Google and Mozilla.
  - isaacremuant 3y ago
    Customization and freedom means Firefox is always the best for me. How's the safari world of add-on's? Do you have tab containers? What about mobile? Can you block ads?
  - NicoJuicy 3y ago
    On Mac? Probably because only Safari has access to the private api's

IainIreland 3y ago

To be clear: we track it in the sense that we still have it running as part of old dashboards. It's cheap, and every so often it catches an unintended regression. The SpiderMonkey team is very much on board with ignoring microbenchmarks. We care about Speedometer, because it correlates reasonably well with real-world performance. We pick and choose which parts of Jetstream2 we bother with. Anything older than that is strictly a measuring stick, rather than a target: it's nice when the numbers go up, but it will only happen as a side-effect of work we're doing for other reasons. Notably, all the SunSpider wins on this graph are side-effects of Speedometer work we've been doing this year.

titzer 3y ago
I understand the logic, but I never understood why we (and I mean, collectively, the JS VM engineers across the browser vendors) didn't just rewrite the benchmark (or fix it) and name it SunSpider 2. I suggested this many times. It's not that much code.
- londons_explore 3y ago
  Presumably because replicating the mix of operations used in real react/jquery code all over the web accurately isn't trivial...
- IainIreland 3y ago
  Yeah, I've been making this argument for years too. Speedometer 3 is in the works, and it's moving in this direction. On multiple occasions we've done performance analysis of the latest draft version, noticed a particular chunk of code is hot, considered what it would take to optimize it, and then changed it because we don't think it reflects real-world usage. EDIT: And I guess the answer for "why not fix SunSpider specifically?" is that once you've done the work to turn it into something worth optimizing, it no longer bears any resemblance to the original. It probably looks a lot more like Speedometer, honestly.
  - titzer 3y ago
    > EDIT: And I guess the answer for "why not fix SunSpider specifically?" is that once you've done the work to turn it into something worth optimizing, it no longer bears any resemblance to the original. It probably looks a lot more like Speedometer, honestly. I get that, yeah, but superseding the old name is probably worth it though; the internet is full of people who never get the memo except via version numbers :) I tongue-in-cheek proposed that we match the Sunspider ASTs in V8 and either make it infinitely fast or infinitely slow. Now that will kill a benchmark once and for all!