5 ms·

What I'd like to see are benchmarks that involve visiting some sample of sites out of the Alexa top 100 and performing standard tasks (reading Gmail, finding a

by cwyers 11y ago

What I'd like to see are benchmarks that involve visiting some sample of sites out of the Alexa top 100 and performing standard tasks (reading Gmail, finding a certain thing on Amazon.com, etc.) I know that's dependent on things like network conditions, so you'd have to run it a lot of times to get meaningful results, but it'd still provide a more meaningful result than a lot of synthetic benchmarks.

pcwalton 11y ago
I agree that major improvements are needed. SunSpider, Octane, and Kraken are all pure JavaScript benchmarks, very microbenchmarky (look at bitwise-and in the SunSpider suite). Oort Online is a benchmark of WebGL performance—WebGL is definitely worth benchmarking, but how common is it in people's daily browsing? HTML5Test tests whether a bunch of "HTML5" features are sorta present, not whether they actually work or anything—it encourages broken implementations that support just enough to pass the test. The state of browser benchmarking is pretty bad. Right now the common suites of benchmarks overwhelmingly focus on what is easy to measure, not what reflects real-world perception of browser performance.
- Klathmon 11y ago
  I hate the HTML5Test (and alternative) sites with a passion for the exact reason you say. HTML5Test lists iOS 8 as having IndexedDB support but it's so broken it's unusable on that platform. Even Mozilla completely disabled it in their LocalForage library due to it's issues.
  - cozuya 11y ago
    LocalForage sounds like a startup for Portland millennials.
  - justwannasing 11y ago
    HTML5test is OK but caniuse.com shows more but people will complain about every browser test site since I started doing web dev 11 years ago. Here's the thing. Edge is IE without the legacy baggage and that's all it is. No different. It's still behind every other browser in its ability to comply with standards and APIs. Despite Microsoft's claim that it will be an "evergreen" browser, that it will update more often, we have not seen this so the proof needs to be done.
    - jongalloway2 11y ago
      I prefer caniuse.com too. The bit about Edge just being IE without the legacy baggage is incorrect. You can see the new standards and APIs they've either added or are adding here: http://dev.modern.ie/platform/status/?filter=f06000000 http://dev.modern.ie/platform/status/?filter=f06000000 The thing I'm happiest about is the user-agent change so poorly written JavaScript / CSS that was written for webkit instead of standards will now magically start working.
      - justwannasing 11y ago
        The new standards and API support are the same that would have been added to IE12. It doesn't matter that Edge came along.
- cwyers 11y ago
  It feels to me like current benchmarks all measure execution time, not user time. Let's say you go from a reasonably sized web page to another, and then hit the back button. How long to load the previous page? Right click on a link and select open in new tab. How long does it take to open a new tab? It's definitely part of the user's perception of the browser's speed, but I don't think stuff like SunSpider and Octane even tries to measure it.
  - ripberge 11y ago
    Good point. I have a surface and want to use IE 11 more because its touch screen friendly, however the back button performance is slow sometimes. Can be very frustrating, so I typically use Chrome. Hopefully Edge fixes that.
  - pcwalton 11y ago
    Heavily related to this, one key insight that I think has been neglected is that JavaScript benchmarks do a bad job of measuring cold code. Most code on the Web is cold, because much Web browsing consists of fresh page loads. Heavily optimizing JITs are exactly what you don't want for cold code: time spent compiling is time you could have been spending running the code. But benchmarks consist of running the same code repeatedly in a long-running loop, which encourages development of sophisticated, highly-optimizing JITs that spend a lot of time in up-front compilation, which end up being totally pointless for JS whose sole purpose is to run once to place an ad somewhere on the page. It's easy to forget that JavaScript is still primarily a scripting language used to perform high-level application tasks. It's awesome that JS has gone so far low-level, and we should continue to push it as far as it can, but I think the bread and butter that users care about is not so much heavy number crunching but rather quick cold code that needs to run once or twice and get out of the way.
    - jgraham 11y ago
      Related: http://research.microsoft.com/apps/pubs/default.aspx?id=118663 http://research.microsoft.com/apps/pubs/default.aspx?id=1186... I don't know if it's a coincidence that around the time that paper was released Microsoft were considerably behind the curve in terms of js engine performance. The paper also misses the fact that excellent hot-code performance opens up new frontiers for the web like games, photo editing, and similar compute-intensive applications that are not well represented by their sample of popular sites since those are, by definition, limited by what was performant in extant browsers.
deleted 11y ago
[deleted]
arsa 11y ago
I really suggest you look here: https://channel9.msdn.com/Shows/Going+Deep/Lars-Bak-and-Steve-Lucco-Chakra-V8-JavaScript-Open-Source https://channel9.msdn.com/Shows/Going+Deep/Lars-Bak-and-Stev... They do talk about the chicken and egg situation, where top sites will optimize for engines, then engines optimize for sites... i.e. it highly depends on the sites - of course all browsers test on top sites and optimize for them. But then the cycle repeats.

by cwyers 11y ago

pcwalton 11y ago

I agree that major improvements are needed. SunSpider, Octane, and Kraken are all pure JavaScript benchmarks, very microbenchmarky (look at bitwise-and in the SunSpider suite). Oort Online is a benchmark of WebGL performance—WebGL is definitely worth benchmarking, but how common is it in people's daily browsing? HTML5Test tests whether a bunch of "HTML5" features are sorta present, not whether they actually work or anything—it encourages broken implementations that support just enough to pass the test. The state of browser benchmarking is pretty bad. Right now the common suites of benchmarks overwhelmingly focus on what is easy to measure, not what reflects real-world perception of browser performance.

Klathmon 11y ago
I hate the HTML5Test (and alternative) sites with a passion for the exact reason you say. HTML5Test lists iOS 8 as having IndexedDB support but it's so broken it's unusable on that platform. Even Mozilla completely disabled it in their LocalForage library due to it's issues.
- cozuya 11y ago
  LocalForage sounds like a startup for Portland millennials.
- justwannasing 11y ago
  HTML5test is OK but caniuse.com shows more but people will complain about every browser test site since I started doing web dev 11 years ago. Here's the thing. Edge is IE without the legacy baggage and that's all it is. No different. It's still behind every other browser in its ability to comply with standards and APIs. Despite Microsoft's claim that it will be an "evergreen" browser, that it will update more often, we have not seen this so the proof needs to be done.
  - jongalloway2 11y ago
    I prefer caniuse.com too. The bit about Edge just being IE without the legacy baggage is incorrect. You can see the new standards and APIs they've either added or are adding here: http://dev.modern.ie/platform/status/?filter=f06000000 http://dev.modern.ie/platform/status/?filter=f06000000 The thing I'm happiest about is the user-agent change so poorly written JavaScript / CSS that was written for webkit instead of standards will now magically start working.
    - justwannasing 11y ago
      The new standards and API support are the same that would have been added to IE12. It doesn't matter that Edge came along.
cwyers 11y ago
It feels to me like current benchmarks all measure execution time, not user time. Let's say you go from a reasonably sized web page to another, and then hit the back button. How long to load the previous page? Right click on a link and select open in new tab. How long does it take to open a new tab? It's definitely part of the user's perception of the browser's speed, but I don't think stuff like SunSpider and Octane even tries to measure it.
- ripberge 11y ago
  Good point. I have a surface and want to use IE 11 more because its touch screen friendly, however the back button performance is slow sometimes. Can be very frustrating, so I typically use Chrome. Hopefully Edge fixes that.
- pcwalton 11y ago
  Heavily related to this, one key insight that I think has been neglected is that JavaScript benchmarks do a bad job of measuring cold code. Most code on the Web is cold, because much Web browsing consists of fresh page loads. Heavily optimizing JITs are exactly what you don't want for cold code: time spent compiling is time you could have been spending running the code. But benchmarks consist of running the same code repeatedly in a long-running loop, which encourages development of sophisticated, highly-optimizing JITs that spend a lot of time in up-front compilation, which end up being totally pointless for JS whose sole purpose is to run once to place an ad somewhere on the page. It's easy to forget that JavaScript is still primarily a scripting language used to perform high-level application tasks. It's awesome that JS has gone so far low-level, and we should continue to push it as far as it can, but I think the bread and butter that users care about is not so much heavy number crunching but rather quick cold code that needs to run once or twice and get out of the way.
  - jgraham 11y ago
    Related: http://research.microsoft.com/apps/pubs/default.aspx?id=118663 http://research.microsoft.com/apps/pubs/default.aspx?id=1186... I don't know if it's a coincidence that around the time that paper was released Microsoft were considerably behind the curve in terms of js engine performance. The paper also misses the fact that excellent hot-code performance opens up new frontiers for the web like games, photo editing, and similar compute-intensive applications that are not well represented by their sample of popular sites since those are, by definition, limited by what was performant in extant browsers.

deleted 11y ago

[deleted]

arsa 11y ago

I really suggest you look here: https://channel9.msdn.com/Shows/Going+Deep/Lars-Bak-and-Steve-Lucco-Chakra-V8-JavaScript-Open-Source https://channel9.msdn.com/Shows/Going+Deep/Lars-Bak-and-Stev... They do talk about the chicken and egg situation, where top sites will optimize for engines, then engines optimize for sites... i.e. it highly depends on the sites - of course all browsers test on top sites and optimize for them. But then the cycle repeats.