7 ms·
Google Criticizes JavaScript Benchmarks, Modifies Sunspider
- js2 16y agohttp://blog.chromium.org/2011/05/updating-javascript-benchmarks-for.html http://blog.chromium.org/2011/05/updating-javascript-benchma...
- kalleboo 16y agoI was hoping they'd discuss how SunSpider doesn't test the DOM, Canvas or other things that web apps commonly do. I haven't felt my apps have really been performance constrained by basic JS language performance, but rather DOM tree searches, element creation, innerHTML parsing, etc. Benchmarking a JavaScript Raytracer and encryption engine is a noble cause for advancing JS going forward (especially with projects such as node.js gaining momentum), but it doesn't feel very relevant to me, right now, today. What other benchmarks are there? Apple's Sunspider, Google's V8 Benchmark Suite, Mozilla's Kraken, that JSLint-based one... Does anyone know of a good link where someone has deconstructed all these JS benchmarks and what parts of a browser they actually test?
- kenjackson 16y agoYes, running a benchmark that doesn't test the right things 50x more doesn't really fix the core issue. I am curious though why IE gets slower. At worst it should run about the same speed.
- andrewcooke 16y agopostponed gc?
- mbelshe 16y agoThe great thing and the worst thing about benchmarks is that vendors optimize to them :-) IE9 claims speed, but the only benchmark it performs better than other browsers on is the SunSpider benchmark. If it is so fast, why does it only do well on a single benchmark? It has already been documented that trivial changes to SS make IE9's performance change dramatically: http://blog.mozilla.com/rob-sayre/2010/11/16/reporting-a-bug-on-a-fragile-analysis/ http://blog.mozilla.com/rob-sayre/2010/11/16/reporting-a-bug... Personally, I believe SunSpider is just too-easy-to-game. Historically, this problem comes up now and then with compiler vendors and benchmarks. The only solution is to have long-running benchmarks which exercise many parts of the JS engine so that gaming is not so easy.
- kenjackson 16y agoActually the IE9 SS issue was a bug. It's since been fixed and no longer displays this weird behavior in IE9. IE9 also does well on other benchmarks too like the Facebook graphics benchmark, where I think the only browser to beat it was FF4.
- mbelshe 16y agoThere were multiple issues. There was a straight up bug, which was fixed. But the perf delta remains.
- kenjackson 16y agoI don't believe so. I had checked this and the perf issue went away. Maybe you're referring to a different issue. Can you post a link with perf data?
- mbelshe 16y agoYou can see this for yourself - run the test referenced in Rob's post above. IE9, right now, reports a 13-14x slowdown on that test with trivial changes.
- rimantas 16y agoTesting pure JS without DOM operations is the reason why I don't pay that much attention to all the bragging about speed: it's mostly irrelevant to what I do. There is however http://dromaeo.com/ http://dromaeo.com/ which does test for DOM operations.
- othermaciej 16y agoSunSpider was deliberately designed to test JS only. What's regrettable is that folks started interpreting JS performance as the primary or even sole metric of browser performance. JS performance has improved by more than an order of magnitude in most browsers since JS was first created, so it's even less likely to be a bottleneck. It's tricky to make good benchmarks. One interest area of new browser-based benchmarks is ones that focus on graphics. Here are some recent benchmarks that focus on graphics, which I think is an up and coming hot area for web applications, particularly games: JSGameBench - http://www.facebook.com/note.php?note_id=491691753919&id=9445547199 http://www.facebook.com/note.php?note_id=491691753919&id... MainInBlue AnimationBenchmark - http://www.themaninblue.com/writing/perspective/2010/03/22/ http://www.themaninblue.com/writing/perspective/2010/03/22/ GUIMark2 - http://www.craftymind.com/guimark2/ http://www.craftymind.com/guimark2/
- justincormack 16y agoSo you run the tests a lot of times so the JIT compiler kicks in and is guaranteed to compile the code. I can see that makes it run faster, but not sure it makes it any more of a real benchmark...
- azakai 16y agoThere are several inaccuracies in this article, * "Mozilla is tinkering with a SpiderMonkey JavaScript engine that supports Google’s V8 core and is currently developed as “V8Monkey”" - I assume they mean the side project some people are doing, to let SpiderMonkey support the V8 API, which would let it be used say in node.js. The article implies that that is the latest "buzz" in JS engines, but there are plenty of actual recent stories in that area, and all major JS engines are constantly improving. * "Mozilla Firefox 5 Beta Build 1" - No such thing exists. Firefox 5 is in Aurora, which is pre-beta, and there is no Firefox Beta at the moment (just nightly, aurora, and stable). * They use Chromium nightly, but not a Firefox nightly (Firefox 6), which is odd. For the other browsers - Opera and IE - I don't know if nightly builds are available, but for Firefox they are. As to the content, I agree with Google that it is interesting to see 'steady-state' results, of running a benchmark many times. For some web pages, that clearly matters. However, the normal SunSpider is still very interesting - it says something about the speed of smaller web pages that try to load quickly, but still need some amount of JavaScript. Quick loading and responsiveness are very important there, and in fact that is exactly the sort of thing Google usually stresses in its own websites, and ironic that Chrome is the slowest there. But in any case, both benchmarks are interesting.
- abraham 16y agoIt looks like Fx5b1 is available through Mozilla's FTP server. http://www.conceivablytech.com/7132/products/first-firefox-5-beta-build-posted http://www.conceivablytech.com/7132/products/first-firefox-5...
- azakai 16y agoHmm, that article is mistaken as well. A Firefox 5 beta should be available in a few weeks. This is the official page for all Firefox channels: http://www.mozilla.com/en-US/firefox/channel/ http://www.mozilla.com/en-US/firefox/channel/
- magicalist 16y agoironic maybe, but when the difference between the best and worst performers over the entire benchmark is 3 frames at 60fps, it isn't unreasonable to think of sunspider as indicative of not much any more. as js engine improvements are starting to level off, the tradeoffs that have to be made become more complex. optimizing for one case can cause a regression in another (just take a look at the heuristics used for when tracing should kick in for trace-/jaegermonkey). at some point, insisting on monotonically decreasing times for a test that takes a quarter of a second becomes pretty much akin to the megahertz or megapixel races.
- othermaciej 16y agoI'm pretty surprised that Google chose to fork SunSpider without even suggesting their change to the maintainer (me) first, let alone providing a patch. That's not to say I necessarily agree that running each test 50 times is a better test, but it seems somewhat outside open source norms of courtesy to fork and then blog about it without even proposing your change first.
- scott_s 16y agoI think that what they did is reasonable. I don't think even they think that running each test 50 times is best ("While repeating a trivial test many times isn’t a great solution, it does provide an opportunity for some optimization"). I think this is more of an experiment, less of a formal change request. I think it's appropriate to publicly post the results and code behind experiments.
- pbiggar 16y agoI would generally agree, but to any observer it looks like SunSpider isn't being actively developed: http://bit.ly/lScLRi http://bit.ly/lScLRi - my correctness bugs have gone unanswered since August, - other mozillians bugs have not been fixed either (at least one since 2009), - the SunSpider svn logs show that there is no work being done on the sunspider tests themselves, since the release of 0.9.1 in January 2010, - the last fix to the harness was 5 months ago (which broke shell tests for everyone except JSC). If you [1] want to remain the "maintainer" you have to show that's you're maintaining it. While this isn't how I would have gone about it (and I was thinking about it), I can't really fault Google for this. [1] And by "you", I mean "Apple", not you personally.
- sayrer 16y agoThey should still file bugs and ping the maintainers, at least.
- pbiggar 16y agoI agree that would be the right way to do it, and I'm sure that we [1] would have done it that way. There is a certain shoot-first-ask-questions-later approach to forking that, while unpleasant, is certainly effective. There are two ways Google could have gone: 1) Talk to Apple, file bugs, wait around for a year to see if anything is being done. Then threaten to fork, but wait when Apple says they're doing it, but they're really not, they're just trying to avoid a fork. Or worse, Apple would simply refuse, saying that it wasn't a good direction for the SunSpider benchmarks to go, which would be legitimate. Then Google is stuck with worse V8 scores forever. 2) Just fork it. They didn't cite a reason, but when asked they no doubt have a few up their sleeves. Now Apple is on the back foot, doesn't have a plan, and perhaps has lost maintainership of SunSpider forever. From a public perception point of view, this is a real winner for Google. They have better open source cred than Apple [2], and they keep V8 up-to-date pretty well. I would be unsurprised if people started reporting the forked scores from now on. That will probably depend on how they steward the project, for example by doing some trivial maintenance like the bugs we've reported on SunSpider. While I don't agree with this approach, I can see how it achieves their aims better. It's reminiscent of the launch of V8 or Google code, where the surprise is used to spur competition from the incumbents. With luck, Apple will update SunSpider like they suggested they would in 2009, there'll be a merge, everything will be forgiven, and JS benchmarking will be better. [1] for those watching at home, sayrer and I both work on the JS team at Mozilla [3] (he's actually my boss). [2] Google may have a closed development model for the V8 engine, but that's par for the course with Apple. [3] Nothing I say represents Mozilla, etc, etc.
- itissid 16y agoShould these tests not be a black box? If you know these tests you can cheat on them right? So the idea of making a black box benchmark is that a closed system randomly chooses tests to run(without you knowing which ones or there nature) and then running them and just giving you the results as a composite metric to tell you how good the engine was. And put in an extra restriction that you can't test more than once every week(say), or some time frame like that. like that so that no one can test infinitely to game it. Folks into machine learning would know what i am talking about here right?
- natmaster 16y agoWe already knew sunspider was outdated last year: http://weblogs.mozillazine.org/asa/archives/2010/10/some_sunspider_numbe.html http://weblogs.mozillazine.org/asa/archives/2010/10/some_sun... It's funny that Google should criticize a benchmark on 'not reflecting the real world' when their own V8 includes machine translated Scheme (how many websites actually do that?) test. Making the sunspider test run more times doesn't make it any better, all it does is tune the benchmark to look better for their particular JIT settings. They spend more time compiling and optimizing the javascript, which can only be taken advantage of after many runs, rather than being smart and balancing optimization time with how much gains you get from actually running with the optimizations.
- ryanpetrich 16y agoJavaScript as a target language isn't unheard of--Google Web Toolkit compiles Java to JavaScript. You're right in that machine translating Scheme to JavaScript is very uncommon.