9 ms·

IE10: How does it really stack up?

93 points by pughbri 14y ago

emehrkay 14y ago
Were all of these tests done on a Windows8 machine? I only ask because the latest version of Safari isnt available for Windows (you may be able to install WebKit nightly as a Windows user though). Id like to see how nightly build of browsers stack up as well.
- bendilts 14y ago
  That's right, these were all done on a Windows 8 machine. We could of course test on Safari 6 on a Mac, but then the numbers wouldn't be directly comparable (since it's different hardware).
  - olliesaunders 14y ago
    I think your article would benefit from making this piont clearer. There's been discussions on HN before about how performance of Apple software on Windows differs greatly compared with performance of Apple software on Apple OSes.
  - thumperama 14y ago
    The same version of Chrome across both platforms can be a control proxy.
ck2 14y ago
We gathered timing information for the parts of the benchmark that it did complete Which parts didn't complete? Did you try the beta or aurora? Code that can consistently crash firefox would probably be elgible for a security bounty.
- bendilts 14y ago
  Interestingly, the part that crashed Firefox was one of the simpler tests--typing a bunch of text content in, then resizing and rotating the shapes that contain that text. Firefox didn't give any details as to why it crashed. The only thing I could even measure that would lead to a crash is memory usage. I just re-ran the test and watched the process's memory usage. It hovered in the 300-400MB range during the first third of the test. Then, during the test I describe, memory usage rocketed to around 1GB, at which point the browser crashed hard. I'm not sure why Firefox would exhibit this behavior when other browsers don't, but we know a few people at Mozilla (they're customers of ours) so we'll probably reach out.
  - carlosreyes 14y ago
    Why not remove the part of the test that crashed it and post results of the rest of the benchmarks? Those are 3 tests out of 16, I would still like to see how it compares in the other benchmark tests. Seeing "We're sorry. Firefox had a problem and crashed." as well as its exclusion from the performance summary made it feel like there was a bit of a bias against Firefox, where even in your conclusion it's stated that it performs "quite well" - that sentence does not reflect on the rest of the article at all.
  - ck2 14y ago
    resizing and rotating the shapes that contain that text Maybe it was hardware acceleration in the particular machine with the particular video driver you use? Can you try the test on an entirely different video setup just to prove it can get through, even if the numbers cannot be compared?
bhanks 14y ago
My skills are a little rough and this is probably more than a company can devote to a browser war study but it would be interesting to see this benchmark run multiple times or as a Monte Carlo simulation. Also - does anyone know what core runs at 50% in IE even when the user is idle? Not surprised to hear it but curious if anyone knows what it is.
- ygra 14y ago
  While idle, the profiler tells me that the function with highest time is yp at line 5198 in chart.js. But no idea whether that's really the culprit.
  - bendilts 14y ago
    Hi, Lucid CTO here. We've spent a lot of time in the IE profiler, and it never appears that any of our Javascript is consuming much CPU at all. There are a handful of functions, like the one you quoted, that poll frequently on a setInterval. But none of them are doing much more than checking a variable and returning almost every time. It's really bizarre.
striderxiii 14y ago
IE10 had to skip a bunch of frames and use half a core to not quite keep up with chrome and roughly match Firefox... Well, I suppose its an improvement still, but they had a pretty low bar to begin with.
- 0xABADC0DA 14y ago
  In IE10 the JavaScript is multithreaded... there's a separate thread for JS parsing, compiling, and garbage collection. As far as I know this is the only browser with a threaded JavaScript VM. So the skipping frames is probably because the rendering can be done independently from running JavaScript, and the core in use is probably some quirk like the benchmark creating a few objects on a timer, causing the GC to run on another thread -- that kind of thing can be fixed pretty easily, either by tweaking the benchmark (if it were publicly available) or by Microsoft fixing IE. I haven't actually used IE10 yet, but I think you'll find this new architecture makes overall browsing more responsive and smoother than Chrome.
monsterix 14y ago
Well, let's agree that IE10 does well whatever it is designed to do well. Cool. Now what? We have an IE10 browser that compares well with Google Chrome, Opera and perhaps Firefox too (I don't know the exact details about your tests that were making Firefox crash as often as you've claimed in your article). So is this the point where Microsoft takes on the real challenge? Will it go beyond its typical high-speed sales pitch for the new browser and attack the more relevant problem that is to really get those IE6/IE7/IE8 and perhaps even IE9 users to adopt IE10 immediately? Just a few days ago, I saw on HN how IE6 has become a political election issue in South Korea! Pushing the lazy and entrenched users out of the hole is a much more important (and harder) problem to solve than to win medals over browser speed and performance tests. Even if tests like these help push the cause of IE10, it would be appropriate to give out all details of the tests and realize why Firefox was 'disagreeing' to your test at all?
- andybak 14y ago
  IE 6, 7 and maybe 8 users are probably running XP so can't upgrade. IE 9 users on Vista are similarly out of luck. I won't be recommending IE 10 even to my clients on Windows 7 or 8 until they have an silent auto-updater and a frequent release cycle. It's the only way we'll stop IE's user base holding back the web.
  - HarshaThota 14y ago
    They do have auto-updates for IE now: http://blogs.windows.com/ie/b/ie/archive/2011/12/15/ie-to-start-automatic-upgrades-across-windows-xp-windows-vista-and-windows-7.aspx http://blogs.windows.com/ie/b/ie/archive/2011/12/15/ie-to-st... But it's not quite there yet because it relies on Windows Update and it only shows the option if you haven't previously declined an upgrade to a new version of IE. But at least it's a start.
  - at-fates-hands 14y ago
    This is the one thing which always mystified me about MS and IE. Why do they continually dead end their browser to a specific OS release? I can't imagine someone in their marketing department actually thought by doing so would get people to upgrade their OS so they could have a more up to date browser.
VengefulCynic 14y ago
There is one more strange thing about IE10 that I should mention. IE9 often maxes out one CPU core when a Lucidchart diagram is open, even when the user is completely idle. IE10 is somewhat better, but on our test machine it still used about 50% of one core when the user is idle. No other browser has any detectable CPU load under the same conditions. Has anyone else seen behavior similar to this in their own javascripting (is that even a word?)? I hadn't been paying special attention to IE's CPU usage but I'd like to think I would have noticed this in any of my own stuff when I was doing browser compatibility testing.
- digitalinfinity 14y ago
  According to http://blogs.msdn.com/b/ie/archive/2012/06/13/advances-in-javascript-performance-in-ie10-and-windows-8.aspx http://blogs.msdn.com/b/ie/archive/2012/06/13/advances-in-ja..., IE has separate GC and JIT threads- if I were to guess, I bet it's the GC
simonster 14y ago
This should really be titled "IE10: How does it really stack up in one benchmark."
- pbhjpbhj 14y ago
  Not just one benchmark, but one apparently completely undocumented benchmark run under Windows 8 on a single laptop. They don't mention how many times the benchmark ran which makes me suspicious that the answer might be one?? Unless they release the benchmarking tool then this result seems pretty worthless to me. >we’ve developed a benchmark that runs inside of the Lucidchart diagramming application // So maybe that line for Firefox could as well read "Lucidchart crashed"?
  - untog 14y ago
    So maybe that line for Firefox could as well read "Lucidchart crashed"? No, I'd say that JS should never crash the browser, no matter what.
    - pbhjpbhj 14y ago
      OK fair enough but if I code something wrongly it doesn't work and fails badly then I'm at least partly to be blame, no? It's moot anyway as it appears the test is proprietary and so can't be investigated further?
paranoiacblack 14y ago
Wait, did you make the other browsers your default before running the tests? I was under the impression that IE is the only first-class browser and the other browsers would work as third-party apps with limitations on things like the amount of processes it can open, etc. I ask because I supremely doubt that Mozilla Firefox couldn't pass your benchmark, although I don't use it myself.
- ygra 14y ago
  There are no limitations on desktop applications in Windows 8 whatsoever.
- keypusher 14y ago
  Do you have any article source for this claim?
SideburnsOfDoom 14y ago
The real problem is that while there will be a new and better Chrome and in a month and a half, and similar for Firefox, we'll have this build of IE10 for 2 years or maybe more. How they compare at the start of the cycle is not typical, how they compare at the midpoint will be more average, how they compare at the end (where IE7 or IE8 is now) is the worst case.
- ZeroGravitas 14y ago
  Yeah, at first I thought, that's impressive that they can catch up so far, even if they fall back again. But then I thought about how much further ahead Firefox and Chrome would be if they'd effectively abandoned their users a couple of years ago and hid in some research lab to plan the next big version. That's a whole lot less testing, bug-fixing, releasing and reworking etc.
- mtgx 14y ago
  Completely agree. IE10 can't even beat Chrome when it's launching once every 2 years. This means Chrome will simply expand the performance gap during this 2 year period. Now if only Chrome for Android had version/feature parity with Chrome for desktop, because it's a full 5 versions behind right now - and it shows. It's not even competitive anymore on mobile because of this huge development gap. Google needs to take Chrome for Android as seriously as they take Chrome for desktop, if not more so.
- Jabbles 14y ago
  Internet Explorer will start automatically updating. http://blogs.windows.com/ie/b/ie/archive/2011/12/15/ie-to-start-automatic-upgrades-across-windows-xp-windows-vista-and-windows-7.aspx http://blogs.windows.com/ie/b/ie/archive/2011/12/15/ie-to-st...
  - polshaw 14y ago
    The question is how often. Adopting the firefox 2/3 model (albeit less nagging a-la-chrome) is a great step forward from the IE6 one we have had up to now, but they really need to match the rapid release schedules of mozilla and google today. At a minimum quarterly updates (features/engine updates, not just security fixes) to stop hindering the web.
olliesaunders 14y ago
Opera used to market itself almost exclusively on how it was faster than any other comparable browser. I just had a look on the Opera website and now it says "Faster & Safer internet" yet it performed worst on this test. As the underdog in the browser market, despite being what has always seemed to me as a pretty nice piece of software, I feel a bit sorry for them. I hope they can do whatever it takes to catch up with Chrome on the performance front.
- bendilts 14y ago
  We were a bit surprised on Opera's performance as well, since it did reasonably well in our last benchmark (vs. Chrome 6, FF4, IE9 beta, etc). http://www.lucidchart.com/blog/2010/09/16/ie9-ff4-beta-in-real-world-benchmark/ http://www.lucidchart.com/blog/2010/09/16/ie9-ff4-beta-in-re...
- pbhjpbhj 14y ago
  >a look on the Opera website and now it says "Faster & Safer internet" yet it performed worst on this test. // Which page were you on, the front page of the website doesn't say that to me? My first instinct would be that they're referring to their "turbo" implementation and their large mobile install base, cf http://www.opera.com/browser/turbo/ http://www.opera.com/browser/turbo/.
  - olliesaunders 14y ago
    > Which page were you on, the front page of the website doesn't say that to me? It's the contents of the <title> tag so it displays on the title bar of the browser you are using to read it with and in the tab if you are using a browser with tabs.
    - pbhjpbhj 14y ago
      Lol, I don't normally have a title bar (turned off for main FF window) and the relevant part was most likely hidden by having a dozen or so tags open. Thanks answering.
- 14y ago
btipling 14y ago
Others have already mentioned the probably slow pace of IE updates. Another thing I'll mention is that testing IE requires Windows, and I haven't worked at a place in years where anyone on my team was using Windows. Years. I've worked at Ask.com, Cloudkick, Rackspace, and now a new startup. Until I can test IE on a Mac, I wont bother to support it unless customers mention something is broken, and they rarely do. Because few use IE.
- ygra 14y ago
  You may have heard of this new thing called “Virtual Machine”.
  - btipling 14y ago
    My development environment runs on a VM. You can't run multiple VMs on a host. I actually didn't want to reply to you because of your sarcasm. Also your comment history is very windows focused.
    - ygra 14y ago
      The only time I ever had problems running multiple VMs at once was when I used two different VMMs to do so. In fact, testing IE on Windows is equally painful as you need the VMs for the older versions anyway (unless you happen to have a few real machines lying around). Depending on your site's demographics you may choose to ignore IE testing, I understand. But often that demographic is rather disconnected from your own development OS so you may not always get around it. Also your comment history is very JS-focused (but how does that matter in any way?).
      - btipling 14y ago
        My comment history is more varied I think. My development environment runs a lot of stuff, so even if running two vm's at once were realistic, it wouldn't be.
zobzu 14y ago
The benchmark is "internal only". How convenient. So basically those are just meaningless pretty graphics.
Yaggo 14y ago
Benchmarking Safari under Windows pretty meaningless.

93 points by pughbri 14y ago

Were all of these tests done on a Windows8 machine? I only ask because the latest version of Safari isnt available for Windows (you may be able to install WebKit nightly as a Windows user though). Id like to see how nightly build of browsers stack up as well.

bendilts 14y ago
That's right, these were all done on a Windows 8 machine. We could of course test on Safari 6 on a Mac, but then the numbers wouldn't be directly comparable (since it's different hardware).
- olliesaunders 14y ago
  I think your article would benefit from making this piont clearer. There's been discussions on HN before about how performance of Apple software on Windows differs greatly compared with performance of Apple software on Apple OSes.
- thumperama 14y ago
  The same version of Chrome across both platforms can be a control proxy.

ck2 14y ago

We gathered timing information for the parts of the benchmark that it did complete Which parts didn't complete? Did you try the beta or aurora? Code that can consistently crash firefox would probably be elgible for a security bounty.

bendilts 14y ago
Interestingly, the part that crashed Firefox was one of the simpler tests--typing a bunch of text content in, then resizing and rotating the shapes that contain that text. Firefox didn't give any details as to why it crashed. The only thing I could even measure that would lead to a crash is memory usage. I just re-ran the test and watched the process's memory usage. It hovered in the 300-400MB range during the first third of the test. Then, during the test I describe, memory usage rocketed to around 1GB, at which point the browser crashed hard. I'm not sure why Firefox would exhibit this behavior when other browsers don't, but we know a few people at Mozilla (they're customers of ours) so we'll probably reach out.
- carlosreyes 14y ago
  Why not remove the part of the test that crashed it and post results of the rest of the benchmarks? Those are 3 tests out of 16, I would still like to see how it compares in the other benchmark tests. Seeing "We're sorry. Firefox had a problem and crashed." as well as its exclusion from the performance summary made it feel like there was a bit of a bias against Firefox, where even in your conclusion it's stated that it performs "quite well" - that sentence does not reflect on the rest of the article at all.
- ck2 14y ago
  resizing and rotating the shapes that contain that text Maybe it was hardware acceleration in the particular machine with the particular video driver you use? Can you try the test on an entirely different video setup just to prove it can get through, even if the numbers cannot be compared?

bhanks 14y ago

My skills are a little rough and this is probably more than a company can devote to a browser war study but it would be interesting to see this benchmark run multiple times or as a Monte Carlo simulation. Also - does anyone know what core runs at 50% in IE even when the user is idle? Not surprised to hear it but curious if anyone knows what it is.

ygra 14y ago
While idle, the profiler tells me that the function with highest time is yp at line 5198 in chart.js. But no idea whether that's really the culprit.
- bendilts 14y ago
  Hi, Lucid CTO here. We've spent a lot of time in the IE profiler, and it never appears that any of our Javascript is consuming much CPU at all. There are a handful of functions, like the one you quoted, that poll frequently on a setInterval. But none of them are doing much more than checking a variable and returning almost every time. It's really bizarre.

striderxiii 14y ago

IE10 had to skip a bunch of frames and use half a core to not quite keep up with chrome and roughly match Firefox... Well, I suppose its an improvement still, but they had a pretty low bar to begin with.

0xABADC0DA 14y ago
In IE10 the JavaScript is multithreaded... there's a separate thread for JS parsing, compiling, and garbage collection. As far as I know this is the only browser with a threaded JavaScript VM. So the skipping frames is probably because the rendering can be done independently from running JavaScript, and the core in use is probably some quirk like the benchmark creating a few objects on a timer, causing the GC to run on another thread -- that kind of thing can be fixed pretty easily, either by tweaking the benchmark (if it were publicly available) or by Microsoft fixing IE. I haven't actually used IE10 yet, but I think you'll find this new architecture makes overall browsing more responsive and smoother than Chrome.

monsterix 14y ago

Well, let's agree that IE10 does well whatever it is designed to do well. Cool. Now what? We have an IE10 browser that compares well with Google Chrome, Opera and perhaps Firefox too (I don't know the exact details about your tests that were making Firefox crash as often as you've claimed in your article). So is this the point where Microsoft takes on the real challenge? Will it go beyond its typical high-speed sales pitch for the new browser and attack the more relevant problem that is to really get those IE6/IE7/IE8 and perhaps even IE9 users to adopt IE10 immediately? Just a few days ago, I saw on HN how IE6 has become a political election issue in South Korea! Pushing the lazy and entrenched users out of the hole is a much more important (and harder) problem to solve than to win medals over browser speed and performance tests. Even if tests like these help push the cause of IE10, it would be appropriate to give out all details of the tests and realize why Firefox was 'disagreeing' to your test at all?

andybak 14y ago
IE 6, 7 and maybe 8 users are probably running XP so can't upgrade. IE 9 users on Vista are similarly out of luck. I won't be recommending IE 10 even to my clients on Windows 7 or 8 until they have an silent auto-updater and a frequent release cycle. It's the only way we'll stop IE's user base holding back the web.
- HarshaThota 14y ago
  They do have auto-updates for IE now: http://blogs.windows.com/ie/b/ie/archive/2011/12/15/ie-to-start-automatic-upgrades-across-windows-xp-windows-vista-and-windows-7.aspx http://blogs.windows.com/ie/b/ie/archive/2011/12/15/ie-to-st... But it's not quite there yet because it relies on Windows Update and it only shows the option if you haven't previously declined an upgrade to a new version of IE. But at least it's a start.
- at-fates-hands 14y ago
  This is the one thing which always mystified me about MS and IE. Why do they continually dead end their browser to a specific OS release? I can't imagine someone in their marketing department actually thought by doing so would get people to upgrade their OS so they could have a more up to date browser.

VengefulCynic 14y ago

There is one more strange thing about IE10 that I should mention. IE9 often maxes out one CPU core when a Lucidchart diagram is open, even when the user is completely idle. IE10 is somewhat better, but on our test machine it still used about 50% of one core when the user is idle. No other browser has any detectable CPU load under the same conditions. Has anyone else seen behavior similar to this in their own javascripting (is that even a word?)? I hadn't been paying special attention to IE's CPU usage but I'd like to think I would have noticed this in any of my own stuff when I was doing browser compatibility testing.

digitalinfinity 14y ago
According to http://blogs.msdn.com/b/ie/archive/2012/06/13/advances-in-javascript-performance-in-ie10-and-windows-8.aspx http://blogs.msdn.com/b/ie/archive/2012/06/13/advances-in-ja..., IE has separate GC and JIT threads- if I were to guess, I bet it's the GC

simonster 14y ago

This should really be titled "IE10: How does it really stack up in one benchmark."

pbhjpbhj 14y ago
Not just one benchmark, but one apparently completely undocumented benchmark run under Windows 8 on a single laptop. They don't mention how many times the benchmark ran which makes me suspicious that the answer might be one?? Unless they release the benchmarking tool then this result seems pretty worthless to me. >we’ve developed a benchmark that runs inside of the Lucidchart diagramming application // So maybe that line for Firefox could as well read "Lucidchart crashed"?
- untog 14y ago
  So maybe that line for Firefox could as well read "Lucidchart crashed"? No, I'd say that JS should never crash the browser, no matter what.
  - pbhjpbhj 14y ago
    OK fair enough but if I code something wrongly it doesn't work and fails badly then I'm at least partly to be blame, no? It's moot anyway as it appears the test is proprietary and so can't be investigated further?

paranoiacblack 14y ago

Wait, did you make the other browsers your default before running the tests? I was under the impression that IE is the only first-class browser and the other browsers would work as third-party apps with limitations on things like the amount of processes it can open, etc. I ask because I supremely doubt that Mozilla Firefox couldn't pass your benchmark, although I don't use it myself.

ygra 14y ago
There are no limitations on desktop applications in Windows 8 whatsoever.
keypusher 14y ago
Do you have any article source for this claim?

SideburnsOfDoom 14y ago

The real problem is that while there will be a new and better Chrome and in a month and a half, and similar for Firefox, we'll have this build of IE10 for 2 years or maybe more. How they compare at the start of the cycle is not typical, how they compare at the midpoint will be more average, how they compare at the end (where IE7 or IE8 is now) is the worst case.

ZeroGravitas 14y ago
Yeah, at first I thought, that's impressive that they can catch up so far, even if they fall back again. But then I thought about how much further ahead Firefox and Chrome would be if they'd effectively abandoned their users a couple of years ago and hid in some research lab to plan the next big version. That's a whole lot less testing, bug-fixing, releasing and reworking etc.
mtgx 14y ago
Completely agree. IE10 can't even beat Chrome when it's launching once every 2 years. This means Chrome will simply expand the performance gap during this 2 year period. Now if only Chrome for Android had version/feature parity with Chrome for desktop, because it's a full 5 versions behind right now - and it shows. It's not even competitive anymore on mobile because of this huge development gap. Google needs to take Chrome for Android as seriously as they take Chrome for desktop, if not more so.
Jabbles 14y ago
Internet Explorer will start automatically updating. http://blogs.windows.com/ie/b/ie/archive/2011/12/15/ie-to-start-automatic-upgrades-across-windows-xp-windows-vista-and-windows-7.aspx http://blogs.windows.com/ie/b/ie/archive/2011/12/15/ie-to-st...
- polshaw 14y ago
  The question is how often. Adopting the firefox 2/3 model (albeit less nagging a-la-chrome) is a great step forward from the IE6 one we have had up to now, but they really need to match the rapid release schedules of mozilla and google today. At a minimum quarterly updates (features/engine updates, not just security fixes) to stop hindering the web.

olliesaunders 14y ago

Opera used to market itself almost exclusively on how it was faster than any other comparable browser. I just had a look on the Opera website and now it says "Faster & Safer internet" yet it performed worst on this test. As the underdog in the browser market, despite being what has always seemed to me as a pretty nice piece of software, I feel a bit sorry for them. I hope they can do whatever it takes to catch up with Chrome on the performance front.

bendilts 14y ago
We were a bit surprised on Opera's performance as well, since it did reasonably well in our last benchmark (vs. Chrome 6, FF4, IE9 beta, etc). http://www.lucidchart.com/blog/2010/09/16/ie9-ff4-beta-in-real-world-benchmark/ http://www.lucidchart.com/blog/2010/09/16/ie9-ff4-beta-in-re...
pbhjpbhj 14y ago
>a look on the Opera website and now it says "Faster & Safer internet" yet it performed worst on this test. // Which page were you on, the front page of the website doesn't say that to me? My first instinct would be that they're referring to their "turbo" implementation and their large mobile install base, cf http://www.opera.com/browser/turbo/ http://www.opera.com/browser/turbo/.
- olliesaunders 14y ago
  > Which page were you on, the front page of the website doesn't say that to me? It's the contents of the <title> tag so it displays on the title bar of the browser you are using to read it with and in the tab if you are using a browser with tabs.
  - pbhjpbhj 14y ago
    Lol, I don't normally have a title bar (turned off for main FF window) and the relevant part was most likely hidden by having a dozen or so tags open. Thanks answering.
14y ago

btipling 14y ago

Others have already mentioned the probably slow pace of IE updates. Another thing I'll mention is that testing IE requires Windows, and I haven't worked at a place in years where anyone on my team was using Windows. Years. I've worked at Ask.com, Cloudkick, Rackspace, and now a new startup. Until I can test IE on a Mac, I wont bother to support it unless customers mention something is broken, and they rarely do. Because few use IE.

ygra 14y ago
You may have heard of this new thing called “Virtual Machine”.
- btipling 14y ago
  My development environment runs on a VM. You can't run multiple VMs on a host. I actually didn't want to reply to you because of your sarcasm. Also your comment history is very windows focused.
  - ygra 14y ago
    The only time I ever had problems running multiple VMs at once was when I used two different VMMs to do so. In fact, testing IE on Windows is equally painful as you need the VMs for the older versions anyway (unless you happen to have a few real machines lying around). Depending on your site's demographics you may choose to ignore IE testing, I understand. But often that demographic is rather disconnected from your own development OS so you may not always get around it. Also your comment history is very JS-focused (but how does that matter in any way?).
    - btipling 14y ago
      My comment history is more varied I think. My development environment runs a lot of stuff, so even if running two vm's at once were realistic, it wouldn't be.

zobzu 14y ago

The benchmark is "internal only". How convenient. So basically those are just meaningless pretty graphics.

Yaggo 14y ago

Benchmarking Safari under Windows pretty meaningless.