47 ms·

Browser Benchmark Battle October 2016: Chrome vs. Firefox vs. Edge

64 points by vezycash 10y ago

lern_too_spel 10y ago
Some plots start the y axis at 0, and some do not. The "Oort Online" plot is especially misleading.
- magicalist 10y ago
  Yeah, these all desperately need to start at 0. The magnitude of the differences in a lot of these really don't matter much.
  - allendoerfer 10y ago
    The WebXPRT test even seems to have a margin of error and Chrome and FF are in each others range. I would like to see what they did to ensure a clean system state and whether they repeated the tests and averaged the results or just showed the first one.
hackuser 10y ago
All the major browsers perform well enough for the great majority of users. For me the differentiator is privacy with regard to the browser vendor and websites I visit, and for that I trust only Mozilla because of their mission and track record (and I use some add-ons). I've read that Chrome is more secure, in terms of integrity when attacked, which of course can help privacy (i.e., confidentiality). I've read that it's due to Chrome's security model, but I'd be interested if someone would be willing to describe it in more detail.
- martey 10y ago
  "For me the differentiator is privacy with regard to the browser vendor and websites I visit, and for that I trust only Mozilla because of their mission and track record (and I use some add-ons)." Are you concerned about Firefox's previous integrations with Pocket [1] and Firefox Hello [2]? [1]: https://groups.google.com/forum/#!msg/mozilla.governance/2PYq2w8tejs/i_IindFDxxgJ https://groups.google.com/forum/#!msg/mozilla.governance/2PY... [2]: https://news.ycombinator.com/item?id=9668526 https://news.ycombinator.com/item?id=9668526
  - hackuser 10y ago
    I don't remember the details of those issues, other than concluding that the risks were way overblown and they were easy to disable.
- mastazi 10y ago
  > For me the differentiator is privacy [...] for that I trust only Mozilla [...] (and I use some add-ons). What FF add-ons and/or configuration do you recommend? I use Privacy Badger, AdBlock Plus and HTTPS Everywhere. In Firefox Options, I clear cookies at the end of every session. If I had Private Mode always active, then I could use Firefox's own Tracking Protection, but I thought Privacy Badger is enough and is made by the EFF which I trust.
  - incompatible 10y ago
    Self-destructing cookies is nice.
  - hackuser 10y ago
    I'm assuming you are an advanced user who can understand and manage these add-ons, and also diagnose the occasional website malfunctions that they cause. Here are some of the most useful I know about: --- * uMatrix: Use this for five minutes and I think you'll be sold on it. Essentially an application-level firewall for your web browser, which replaces several other add-ons. Controls cookies, JavaScript, plugins, frames, provides connection-level privacy blacklists, and all at a granular level, using firewall-style rules for local (to the website) & remote pairs: e.g., * * * DENY # Deny all default cnn.com googleapis.com javascript ALLOW But the most impressive part is the GUI, the "matrix", which allows you to visualize as well as create/remove rules very efficiently, with a click (though you can still write them manually if you like). Like a well-made car, it's so well-designed and executed that it's a pleasure to drive. Real firewalls should use this GUI. --- * NoScript: Does far more than block JavaScript, and provides a bunch of security that uMatrix does not: https://news.ycombinator.com/item?id=12624596 https://news.ycombinator.com/item?id=12624596 --- * Decentraleyes: A local cache of common CDNs and other resources. It protects privacy by minimizing connections to CDNs, should reduce bandwidth, and can help performance in some cases. From their documentation: - Supported Networks: Google Hosted Libraries, Microsoft Ajax CDN, CDNJS (Cloudflare), jQuery CDN (MaxCDN), jsDelivr (MaxCDN), Yandex CDN, Baidu CDN, Sina Public Resources, and UpYun Libraries. - Bundled Resources: AngularJS, Backbone.js, Dojo, Ember.js, Ext Core, jQuery, jQuery UI, Modernizr, MooTools, Prototype, Scriptaculous, SWFObject, Underscore.js, and Web Font Loader. --- * HTTPS Everywhere: For the reference of others reading this comment, it's an EFF project that uses a whitelist of websites which accept https connections and forces the browser to connect to them over the secure protocol. --- * CsFire: CSRF detection and protection. Also technically interesting. https://distrinet.cs.kuleuven.be/software/CsFire/ https://distrinet.cs.kuleuven.be/software/CsFire/
- dingo_bat 10y ago
  >All the major browsers perform well enough for the great majority of users. I guess I'm in the minority then. Firefox is markedly slow in rendering and navigating large pages. Even searching inside the page is slow. Chrome doesn't even notice a difference. Edge is similarly indifferent to the page size, but it's buggy as hell in general. It's infuriating that after all these years Firefox is still so much behind on basic performance.
  - ploxiln 10y ago
    What's infuriating to me (except that I've mellowed about it over the years) is that web pages use as much cpu and memory as they can get away with, up to the edge of un-usability. Pre-mature optimization being evil and all that. FWIW here's a huge page with colored text and links (real html stuff), and it's quite snappy in firefox: https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/diff/?id=v4.7.8&id2=v4.7&context=3&ignorews=0&dt=0 https://git.kernel.org/cgit/linux/kernel/git/stable/linux-st...
- confounded 10y ago
  After moving from Firefox to Chromium about five years ago, I've gradually moved back. Initially for Self Destructing Cookies and NoScript, but the Reader View really is fantastic. I've found a locked down version of FF for reading, and Chromium for webapps works very well.
- ryuuchin 10y ago
  > I've read that Chrome is more secure, in terms of integrity when attacked, which of course can help privacy (i.e., confidentiality). I've read that it's due to Chrome's security model, but I'd be interested if someone would be willing to describe it in more detail. The developer documentation[1] does a pretty good job at describing the security model and the defense in depth that it provides. Edge adopts a similar sandbox model although Chrome's is probably better. Their sandbox model allows them to leverage nearly all of the OS mitigations and security features that Windows[2] (and other's) have to offer. One example of a major difference that can be enabled because of Chrome's model is the usage of Win32k.sys lockdown which is the only way which you can reduce the kernel attack surface on Windows. This is enabled for all renderer processes and may be enabled on plugin processes as well (not sure if that got enabled globally yet outside of field tests). Win32k lockdown blocks access to the entire GDI subsystem among some other things. As far as I know Chrome and possibly chromium based browsers are the only ones who currently use the win32k lockdown mitigation. [1] https://www.chromium.org/developers/design-documents/sandbox https://www.chromium.org/developers/design-documents/sandbox [2] https://www.chromium.org/developers/design-documents/sandbox#TOC-Sandbox-restrictions https://www.chromium.org/developers/design-documents/sandbox...
DonCullen 10y ago
Would have been good to see Safari and Opera in the assessment too.
- LeoNatan25 10y ago
  Well, modern Opera is very similar to Chrome, as it is Blink based. Agree about Safari.
- mastazi 10y ago
  As it was mentioned in another comment, you either leave out Safari or Edge, since there is no OS where both are available. Obviously, in order to obtain a meaningful comparison, you need to run all your tests on same OS and same HW specs.
  - matt4077 10y ago
    You could easily use the same hardware, and maybe run Chrome in both OS – giving you a shared reference point. It'd also make the comparison useful to people using OS X. Edit: I just ran the JetStream benchmark on OS X (El Cap). Results: - Safari 10.1: 306.25 ± 5.8662 - Chrome 56: 198.77 ± 10.059 - Firefox 49: 234.56 ± 19.496 (ran it a second time after update: 235.04 ± 6.0682) Their results: - Chrome: 184 ± 2 - Firefox: 154 ± 4 - Edge: 219 ± 5
    - mastazi 10y ago
      Yes, this would be interesting indeed.
pragmatic 10y ago
Edge may be fast in benchmarks but in real life it's a buggy mess. Web pages look terrible and "fuzzy". Local testing with visual studio is broken. It works poorly with office 360 and bing maps. This is just my experience. Only safari comes close to being a big a pain in the ass.
- bwat48 10y ago
  edge's interface looks like it was designed in an intern's spare time
- dingo_bat 10y ago
  To be honest, all metro style apps have the problems you described. Also, there are minor oddities that bug me. For example, while switching desktops, the edge window will appear blank for a split second before filling up with the content. This happens every time. I don't think it has anything to do with memory.
  - agumonkey 10y ago
    Latest Win10 update replaced the permission dialog box with the metro equivalent, keyboard shortcut lost, I cried a little inside.
- jccalhoun 10y ago
  since edge has gotten lastpass and adblock, I have been using it more and more. I haven't noticed pages looking fuzzy. it seems to work well but the fact that you can't right click and search in a new tab is frustrating.
  - ryuuchin 10y ago
    > since edge has gotten lastpass and adblock There's also an experimental version of uBlock for it as well[1]. [1] https://github.com/nikrolls/uBlock-Edge#microsoft-edge https://github.com/nikrolls/uBlock-Edge#microsoft-edge
    - 10y ago
tambourine_man 10y ago
Sunspider was developed by WebKit and yet its browser is not featured on the benchmark
- TheCoreh 10y ago
  There's no version of Safari for Windows, or of Edge for Mac. So you need to leave one of these out when comparing on the same hardware/os combo.
  - frik 10y ago
    Chrome and Safari are used by most users (on mobile and notebook/PC). Both should be included in such benchmarks. Firefox is kind of popular on desktop, and IE11 is still pretty common if you look at corporate users.
    - wolfgke 10y ago
      > Chrome and Safari are used by most users (on mobile and notebook/PC). Statistics for Notebook/PC: > http://gs.statcounter.com/#desktop-browser-ww-monthly-201509-201609 http://gs.statcounter.com/#desktop-browser-ww-monthly-201509... This shows Safari has only a market share of 4,75%. For mobile: As long as it is not possible to either run Safari under another mobile operating system or Apple allows allocate executable memory on iOS to let other browsers implement JIT there is no meaningful way to compare mobile browsers (in particular Safari),
qwertyuiop924 10y ago
FF's JS perf is just embarassing. Hopefully, when the new optimizations land (they're working on some now, IIRC) this will be improved.
deleted 10y ago
[deleted]
cpeterso 10y ago
Microsoft has been accused of "cheating" on SunSpider before because IE/Edge's dead-code elimination optimizations recognizes particular code patterns in SunSpider: http://arstechnica.com/information-technology/2010/11/lies-damned-lies-and-benchmarks-is-ie9-cheating-at-sunspider/ http://arstechnica.com/information-technology/2010/11/lies-d... https://news.ycombinator.com/item?id=1913102 https://news.ycombinator.com/item?id=1913102
pmontra 10y ago
Be sure to look at the numbers in the charts. Those bars there don't use the full axis and suffer from distorsion. It's the #1 don't in the list at http://www.eea.europa.eu/data-and-maps/daviz/learn-more/chart-dos-and-donts http://www.eea.europa.eu/data-and-maps/daviz/learn-more/char...
- lomnakkus 10y ago
  Jebus, you weren't kidding... that's a lot of distortion. In one example we're talking a visual difference of >50% which amounts to an actual difference of 80 "points" out of 10000, so <1%. (No, that's not a typo on my part.) You have to wonder of most of the visual difference isn't actually just statistical noise at that point.
- passive 10y ago
  Came here to post just this. Really ridiculous, and sometimes the text even matches the distorted scale, saying things like "blew past the others" for a 1% win.
- mthoms 10y ago
  I cringed at the words "Firefox left it in the dust" (in comparison to Chrome for the Oort WebGL Benchmark). The actual scores were: Firefox: 10,000 Chrome: 9,940 Hyperbole much?
Turbots 10y ago
FUCK THOSE RETARDED SCALES ON THE CHARTS!
andrewclunn 10y ago
No css variable support = no thank you edge.
- Zachery 10y ago
  Soon: https://developer.microsoft.com/en-us/microsoft-edge/platform/status/csscustompropertiesakacssvariables/ https://developer.microsoft.com/en-us/microsoft-edge/platfor...
ry_ry 10y ago
The Edge JS performance is interesting for me wrt serverside rendering - previous benchmarks of the engines running on node (rather than the browser) I've seen have had V8 edging ChakraCore out on Octane circa May this year. If this still the case? The relative Chakra performance looks pretty strong here.
r3dn3r 10y ago
this guys work for Microsoft..... so predictable...

64 points by vezycash 10y ago

lern_too_spel 10y ago

Some plots start the y axis at 0, and some do not. The "Oort Online" plot is especially misleading.

magicalist 10y ago
Yeah, these all desperately need to start at 0. The magnitude of the differences in a lot of these really don't matter much.
- allendoerfer 10y ago
  The WebXPRT test even seems to have a margin of error and Chrome and FF are in each others range. I would like to see what they did to ensure a clean system state and whether they repeated the tests and averaged the results or just showed the first one.

hackuser 10y ago

All the major browsers perform well enough for the great majority of users. For me the differentiator is privacy with regard to the browser vendor and websites I visit, and for that I trust only Mozilla because of their mission and track record (and I use some add-ons). I've read that Chrome is more secure, in terms of integrity when attacked, which of course can help privacy (i.e., confidentiality). I've read that it's due to Chrome's security model, but I'd be interested if someone would be willing to describe it in more detail.

martey 10y ago
"For me the differentiator is privacy with regard to the browser vendor and websites I visit, and for that I trust only Mozilla because of their mission and track record (and I use some add-ons)." Are you concerned about Firefox's previous integrations with Pocket [1] and Firefox Hello [2]? [1]: https://groups.google.com/forum/#!msg/mozilla.governance/2PYq2w8tejs/i_IindFDxxgJ https://groups.google.com/forum/#!msg/mozilla.governance/2PY... [2]: https://news.ycombinator.com/item?id=9668526 https://news.ycombinator.com/item?id=9668526
- hackuser 10y ago
  I don't remember the details of those issues, other than concluding that the risks were way overblown and they were easy to disable.
mastazi 10y ago
> For me the differentiator is privacy [...] for that I trust only Mozilla [...] (and I use some add-ons). What FF add-ons and/or configuration do you recommend? I use Privacy Badger, AdBlock Plus and HTTPS Everywhere. In Firefox Options, I clear cookies at the end of every session. If I had Private Mode always active, then I could use Firefox's own Tracking Protection, but I thought Privacy Badger is enough and is made by the EFF which I trust.
- incompatible 10y ago
  Self-destructing cookies is nice.
- hackuser 10y ago
  I'm assuming you are an advanced user who can understand and manage these add-ons, and also diagnose the occasional website malfunctions that they cause. Here are some of the most useful I know about: --- * uMatrix: Use this for five minutes and I think you'll be sold on it. Essentially an application-level firewall for your web browser, which replaces several other add-ons. Controls cookies, JavaScript, plugins, frames, provides connection-level privacy blacklists, and all at a granular level, using firewall-style rules for local (to the website) & remote pairs: e.g., * * * DENY # Deny all default cnn.com googleapis.com javascript ALLOW But the most impressive part is the GUI, the "matrix", which allows you to visualize as well as create/remove rules very efficiently, with a click (though you can still write them manually if you like). Like a well-made car, it's so well-designed and executed that it's a pleasure to drive. Real firewalls should use this GUI. --- * NoScript: Does far more than block JavaScript, and provides a bunch of security that uMatrix does not: https://news.ycombinator.com/item?id=12624596 https://news.ycombinator.com/item?id=12624596 --- * Decentraleyes: A local cache of common CDNs and other resources. It protects privacy by minimizing connections to CDNs, should reduce bandwidth, and can help performance in some cases. From their documentation: - Supported Networks: Google Hosted Libraries, Microsoft Ajax CDN, CDNJS (Cloudflare), jQuery CDN (MaxCDN), jsDelivr (MaxCDN), Yandex CDN, Baidu CDN, Sina Public Resources, and UpYun Libraries. - Bundled Resources: AngularJS, Backbone.js, Dojo, Ember.js, Ext Core, jQuery, jQuery UI, Modernizr, MooTools, Prototype, Scriptaculous, SWFObject, Underscore.js, and Web Font Loader. --- * HTTPS Everywhere: For the reference of others reading this comment, it's an EFF project that uses a whitelist of websites which accept https connections and forces the browser to connect to them over the secure protocol. --- * CsFire: CSRF detection and protection. Also technically interesting. https://distrinet.cs.kuleuven.be/software/CsFire/ https://distrinet.cs.kuleuven.be/software/CsFire/
dingo_bat 10y ago
>All the major browsers perform well enough for the great majority of users. I guess I'm in the minority then. Firefox is markedly slow in rendering and navigating large pages. Even searching inside the page is slow. Chrome doesn't even notice a difference. Edge is similarly indifferent to the page size, but it's buggy as hell in general. It's infuriating that after all these years Firefox is still so much behind on basic performance.
- ploxiln 10y ago
  What's infuriating to me (except that I've mellowed about it over the years) is that web pages use as much cpu and memory as they can get away with, up to the edge of un-usability. Pre-mature optimization being evil and all that. FWIW here's a huge page with colored text and links (real html stuff), and it's quite snappy in firefox: https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/diff/?id=v4.7.8&id2=v4.7&context=3&ignorews=0&dt=0 https://git.kernel.org/cgit/linux/kernel/git/stable/linux-st...
confounded 10y ago
After moving from Firefox to Chromium about five years ago, I've gradually moved back. Initially for Self Destructing Cookies and NoScript, but the Reader View really is fantastic. I've found a locked down version of FF for reading, and Chromium for webapps works very well.
ryuuchin 10y ago
> I've read that Chrome is more secure, in terms of integrity when attacked, which of course can help privacy (i.e., confidentiality). I've read that it's due to Chrome's security model, but I'd be interested if someone would be willing to describe it in more detail. The developer documentation[1] does a pretty good job at describing the security model and the defense in depth that it provides. Edge adopts a similar sandbox model although Chrome's is probably better. Their sandbox model allows them to leverage nearly all of the OS mitigations and security features that Windows[2] (and other's) have to offer. One example of a major difference that can be enabled because of Chrome's model is the usage of Win32k.sys lockdown which is the only way which you can reduce the kernel attack surface on Windows. This is enabled for all renderer processes and may be enabled on plugin processes as well (not sure if that got enabled globally yet outside of field tests). Win32k lockdown blocks access to the entire GDI subsystem among some other things. As far as I know Chrome and possibly chromium based browsers are the only ones who currently use the win32k lockdown mitigation. [1] https://www.chromium.org/developers/design-documents/sandbox https://www.chromium.org/developers/design-documents/sandbox [2] https://www.chromium.org/developers/design-documents/sandbox#TOC-Sandbox-restrictions https://www.chromium.org/developers/design-documents/sandbox...

DonCullen 10y ago

Would have been good to see Safari and Opera in the assessment too.

LeoNatan25 10y ago
Well, modern Opera is very similar to Chrome, as it is Blink based. Agree about Safari.
mastazi 10y ago
As it was mentioned in another comment, you either leave out Safari or Edge, since there is no OS where both are available. Obviously, in order to obtain a meaningful comparison, you need to run all your tests on same OS and same HW specs.
- matt4077 10y ago
  You could easily use the same hardware, and maybe run Chrome in both OS – giving you a shared reference point. It'd also make the comparison useful to people using OS X. Edit: I just ran the JetStream benchmark on OS X (El Cap). Results: - Safari 10.1: 306.25 ± 5.8662 - Chrome 56: 198.77 ± 10.059 - Firefox 49: 234.56 ± 19.496 (ran it a second time after update: 235.04 ± 6.0682) Their results: - Chrome: 184 ± 2 - Firefox: 154 ± 4 - Edge: 219 ± 5
  - mastazi 10y ago
    Yes, this would be interesting indeed.

pragmatic 10y ago

Edge may be fast in benchmarks but in real life it's a buggy mess. Web pages look terrible and "fuzzy". Local testing with visual studio is broken. It works poorly with office 360 and bing maps. This is just my experience. Only safari comes close to being a big a pain in the ass.

bwat48 10y ago
edge's interface looks like it was designed in an intern's spare time
dingo_bat 10y ago
To be honest, all metro style apps have the problems you described. Also, there are minor oddities that bug me. For example, while switching desktops, the edge window will appear blank for a split second before filling up with the content. This happens every time. I don't think it has anything to do with memory.
- agumonkey 10y ago
  Latest Win10 update replaced the permission dialog box with the metro equivalent, keyboard shortcut lost, I cried a little inside.
jccalhoun 10y ago
since edge has gotten lastpass and adblock, I have been using it more and more. I haven't noticed pages looking fuzzy. it seems to work well but the fact that you can't right click and search in a new tab is frustrating.
- ryuuchin 10y ago
  > since edge has gotten lastpass and adblock There's also an experimental version of uBlock for it as well[1]. [1] https://github.com/nikrolls/uBlock-Edge#microsoft-edge https://github.com/nikrolls/uBlock-Edge#microsoft-edge
  - 10y ago

tambourine_man 10y ago

Sunspider was developed by WebKit and yet its browser is not featured on the benchmark

TheCoreh 10y ago
There's no version of Safari for Windows, or of Edge for Mac. So you need to leave one of these out when comparing on the same hardware/os combo.
- frik 10y ago
  Chrome and Safari are used by most users (on mobile and notebook/PC). Both should be included in such benchmarks. Firefox is kind of popular on desktop, and IE11 is still pretty common if you look at corporate users.
  - wolfgke 10y ago
    > Chrome and Safari are used by most users (on mobile and notebook/PC). Statistics for Notebook/PC: > http://gs.statcounter.com/#desktop-browser-ww-monthly-201509-201609 http://gs.statcounter.com/#desktop-browser-ww-monthly-201509... This shows Safari has only a market share of 4,75%. For mobile: As long as it is not possible to either run Safari under another mobile operating system or Apple allows allocate executable memory on iOS to let other browsers implement JIT there is no meaningful way to compare mobile browsers (in particular Safari),

qwertyuiop924 10y ago

FF's JS perf is just embarassing. Hopefully, when the new optimizations land (they're working on some now, IIRC) this will be improved.

deleted 10y ago

[deleted]

cpeterso 10y ago

Microsoft has been accused of "cheating" on SunSpider before because IE/Edge's dead-code elimination optimizations recognizes particular code patterns in SunSpider: http://arstechnica.com/information-technology/2010/11/lies-damned-lies-and-benchmarks-is-ie9-cheating-at-sunspider/ http://arstechnica.com/information-technology/2010/11/lies-d... https://news.ycombinator.com/item?id=1913102 https://news.ycombinator.com/item?id=1913102

pmontra 10y ago

Be sure to look at the numbers in the charts. Those bars there don't use the full axis and suffer from distorsion. It's the #1 don't in the list at http://www.eea.europa.eu/data-and-maps/daviz/learn-more/chart-dos-and-donts http://www.eea.europa.eu/data-and-maps/daviz/learn-more/char...

lomnakkus 10y ago
Jebus, you weren't kidding... that's a lot of distortion. In one example we're talking a visual difference of >50% which amounts to an actual difference of 80 "points" out of 10000, so <1%. (No, that's not a typo on my part.) You have to wonder of most of the visual difference isn't actually just statistical noise at that point.
passive 10y ago
Came here to post just this. Really ridiculous, and sometimes the text even matches the distorted scale, saying things like "blew past the others" for a 1% win.
mthoms 10y ago
I cringed at the words "Firefox left it in the dust" (in comparison to Chrome for the Oort WebGL Benchmark). The actual scores were: Firefox: 10,000 Chrome: 9,940 Hyperbole much?

Turbots 10y ago

FUCK THOSE RETARDED SCALES ON THE CHARTS!

andrewclunn 10y ago

No css variable support = no thank you edge.

Zachery 10y ago
Soon: https://developer.microsoft.com/en-us/microsoft-edge/platform/status/csscustompropertiesakacssvariables/ https://developer.microsoft.com/en-us/microsoft-edge/platfor...

ry_ry 10y ago

The Edge JS performance is interesting for me wrt serverside rendering - previous benchmarks of the engines running on node (rather than the browser) I've seen have had V8 edging ChakraCore out on Octane circa May this year. If this still the case? The relative Chakra performance looks pretty strong here.

r3dn3r 10y ago

this guys work for Microsoft..... so predictable...