I'm really surprised that no one in these comments and the thread on the actual story[1] (instead of this...whatever the equivalent of blogspam is but for slash

I'm really surprised that no one in these comments and the thread on the actual story[1] (instead of this...whatever the equivalent of blogspam is but for slashdot) hasn't mentioned the horrible averaging methodology of this benchmark suite. The individual tests aren't even a problem (though I would maybe pick some more and/or different ones, especially their somewhat odd benchmark choices in performance and graphics), but the averaging makes no sense at all. Averaging time-based benchmarks is problematic enough (it feels more right, but it still isn't a good idea), but how on earth do you convert a "number of times we had to refresh a page" result into a number that you can then average with a "standards compliance" count and a measure of memory efficiency?? Even if you normalize (which it doesn't seem like they did from the output numbers, but maybe they did), the numbers still aren't comparable because you've given no account for the relative magnitude of their effect. e.g. if you have a test of "does the browser have a konami code easter egg?", it doesn't matter if the geometric mean is less sensitive to outliers, because it still doesn't make any sense to take an average of that with "the playback framerate of an HD Trololo video" and then pretend that the average provides any insight. And it's even worse if you then compare that average to other crazy averages! At best you can look at relative ranking, which they actually mention but then proceed to give exact numbers for their relative ranking. There's no information about "betterness" in there, though, except if you then divide up the numbers again to say "these points came from the win in test A, and these points from test B"...at which point you just have the original tests again. Better to count win/no-wins and use that as your final result. Then at least it's obvious that if some tests are much more non-trivial than others that you'll have to give arbitrary weights for the final result, as opposed to having the arbitrary weights being implicit in the tests themselves. Sorry for the rant :) This is a good recognition of Mozilla's hard work, though notably they've been winning many of these tests for a while now (especially the memory ones), but it would be nice if tomshardware could drop the basically meaningless overall scores (or we could just collectively ignore them). [1] https://news.ycombinator.com/item?id=5970429 https://news.ycombinator.com/item?id=5970429