7 ms·

Edge platform team employee here. A couple thoughts on methodology, focusing on our scripted test (under "stay productive longer" at https://blogs.windows.com/w

Edge platform team employee here. A couple thoughts on methodology, focusing on our scripted test (under "stay productive longer" at https://blogs.windows.com/windowsexperience/2016/06/20/more-battery-with-edge/ https://blogs.windows.com/windowsexperience/2016/06/20/more-...), which is most comparable to Opera's test; nobody seems to challenge our Netflix results: * We did not enable ad blockers, because we were testing the efficiency of the network stack, browser rendering pipeline, etc.; if you cut off ads, you're effectively skipping half the assignment. This is a valid way to improve the user experience, but not a valid way to measure browser efficiency, especially because it will disproportionately impact some sites (like the news sites that Opera's test focused on) and have no impact on other sites (like Netflix, where Edge demolishes the other browsers). It's basically skipping a lap of the race and then bragging that you finished faster. * It's also worth noting that the ad blockers are often detected by many news sites and will actually prevent the main page content from loading at all. Not sure if Opera's test accounted for this. * Our test was designed to mimic real-world behavior: Watching YouTube (foreground and background), shopping on Amazon, browsing the Facebook news feed, searching on Google, opening email in Gmail, and reading Wikipedia. To reduce variability, we used WebDriver to instrument the tests (supported by all four browsers tested) and made sure each task was timed rather than just a loop of consecutive tasks (which could disadvantage or advantage browsers depending on other factors like pageload performance or network conditions, in a way that doesn't reflect user experience, which is likely to linger on a page once it's loaded). We then used the Maxim 34407 power instrumentation built into the Surface Book (which is why we chose the Surface Book for this test) to measure actual instantaneous power consumption at the hardware, sampled once per second and then averaged across the duration of the test. We feel strongly that this is a highly scientific and defensible test setup which mimics typical user behavior and, significantly, measures the same markup and the same duration, on the same hardware, in every browser.

wernerb 10y ago
The way I read the blog is that the adblocking is enabled to show that even _with_ an expensive extension enabled it still beats Edge, but that was in my opinion not the gist of the post. The post was about that microsoft is not transparent about the methodology/setup/scripts/target websites used. A third party should be able to support Microsoft's claims. Is it possible for the setup to be published to your github.com/microsoft so that it may be executed automatically? Heck, go the extra mile put it in a CI and publish the data on regular basis :)
- IshKebab 10y ago
  Adblock isn't an "expensive extension". Ads are the expensive thing, so blocking them saves power.
  - wernerb 10y ago
    Until we know exactly what websites/videos etc are loaded (also on the Opera benchmark!); you, me and all consumers do not know if this is true :)
  - DominikD 10y ago
    Also Opera has built-in blocking. It's not a JavaScript-heavy extension to begin with. ;)
  - oxymoron 10y ago
    It's not quite that simple [0][1], or at least it wasn't always. AdBlock Plus in particular used to rely on a massive style sheet that was injected into all tabs and frames, which degraded really quickly on pages with a lot of iframes (which, as it happens, is typically pages with lots of ads). I believe Firefox partly resolved the problem on their end later on, although I can't tell you the exact status of things. Nevertheless, there's definitely some precedence for claiming that ad blocking is an expensive operation. Intuitively, you'd think that network based blocking would be enough, but it won't work against same domain ad sources (a typical example being facebook ads), while css selectors are able to capture a bit more depth. Nevertheless, I don't know exactly what has happened since, although I do recall Firefox making some adjustments on their end that improved the situation. [0] http://www.extremetech.com/computing/182428-ironic-iframes-adblock-plus-is-probably-the-reason-firefox-and-chrome-are-such-memory-hogs http://www.extremetech.com/computing/182428-ironic-iframes-a... [1] https://blog.mozilla.org/nnethercote/2014/05/14/adblock-pluss-effect-on-firefoxs-memory-usage/ https://blog.mozilla.org/nnethercote/2014/05/14/adblock-plus...
    - ryuuchin 10y ago
      No, it really is that simple[1]. The only reason I can think of for someone not using uBlock Origin is because they've never heard of it. [1] https://github.com/gorhill/uBlock#performance https://github.com/gorhill/uBlock#performance
spankalee 10y ago
Doesn't the Netflix test really test GPU decoding vs software decoding? I'm not 100% sure on this, so any clarification welcome, but I thought that Netflix content contracts DRM requirements allowed GPU decoding via Playready, but not on some others like Widevine. Would the Netflix test still show such an advantage for Edge if there weren't any DRM?
- j_h_s 10y ago
  It seems to me that if you're running on the same device, the gpu should not be a variable.
  - caconym_ 10y ago
    IIRC everything except Edge and Safari are forced to use Silverlight for Netflix streaming. I'm not exactly sure why this is, but it probably has something to do with the mess that is video codec licensing.
    - jsmith0295 10y ago
      It's for DRM purposes
    - tyleregeto 10y ago
      Netflix plays fine in Chrome (maybe Firefox too) using HTML5, it was opt-in at one point not sure about now. I watch in Linux all the time, no Silverlight there.
      - russjr08 10y ago
        Starting with Firefox 47, it works fine as well (macOS here), but you need to switch your user agent to Chrome (before you go to Netflix). I'm guessing Netflix hasn't updated their compatibility checker.
    - ryuuchin 10y ago
      Chrome uses widevine (DRM) for Netflix which uses HTML5. Really the more significant difference is that Chrome gets capped to 720p. Edge actually gets the 1080p streams (as does IE11 on Windows 8+ and the Netflix Windows Store app).
8_hours_ago 10y ago
It would be interesting to see this test run again with ad blockers enabled on each browser (the latest version of Edge has support for extensions).
hristov 10y ago
So where is your script? Or are we just supposed to trust you?
- kayone 10y ago
  you don't have to do anything.
mynewtb 10y ago
How did you control for the specific ads the browsers were served? Ads are highly dynamic. Unless you (can) include the differences in your methodology you better eliminate that influence.
- patcheudor 10y ago
  Ding, ding, ding and to be fair this is the problem in both methodologies. The only way to accurately perform this test is to spider a bunch of sites, save the contents to a locally hosted HTTPD and ensure all third-party JS calls are resolved locally and test both against the exact same sites as they were spidered at a point in time. You can simply not account for changes which may happen to the markup, ad network beacons, ads, metrics code, or even network routing, all of which could influence the test in one way or the other if not run from an identical static cache in a controlled environment. If I were doing this test I'd then skip the whole scripting automation part and simply add a meta-refresh to every page in the cache to sequentially take the browser through the content, giving each page something like 10 seconds to load and render. Simple, simple, and far more accurate.
  - csl 10y ago
    You don't need to go to such complicated lengths. Just perform enough tests (as in, a statistically large enough amount) and a distribution will form. That also captures the variability of real world network effects.
    - patcheudor 10y ago
      It's hardly complicated. I've put such tests together in an afternoon. In fact, whatever is added in complexity is gained by the fact fewer tests are necessary. Via this mechanism you can also remove any questions about compression, use of HTTP/2, etc., which could impact the tests based on server-side choices when it comes to serving data to either platform. Equal always equals better.
      - dogma1138 10y ago
caconym_ 10y ago
Given that basically everyone should be blocking ads (if for no reason other than that they are a vector for malware), native ad blocking is really an energy saving mode, a user experience improvement, and a crucial security measure all rolled up into one feature. If the focus of this test is meant to be on the user experience, as the tone of the Microsoft blog post seems to suggest, then said native ad blocker is a killer feature that really might mean Opera users can "stay productive longer". If not, then sure, Edge's underlying implementation is probably faster. It would be interesting to see a comparable benchmark of Edge with the best available ad blocking turned on.
hristov 10y ago
Wow this is all fascinating! If it is not too much trouble, would you mind releasing the script that you used in your testing so that independent parties may verify your results.
wdr1 10y ago
Could you disclose/open source your methodology?
Aissen 10y ago
> nobody seems to challenge our Netflix results: Of course, this is Lotus vs Office all over again. You're using the OS advantage (Windows 10 and its DRM platform) to show that you can do things more efficiently. You chose the technology platform (I'm guessing Play Ready), you forced GPU vendors to comply, and now you give competitors a bad reputation for daring to use their own (or a competitor's) technology.
- rkangel 10y ago
  Or, to put it another way: the have built a well integrated technology stack and are now demonstrating that it has superior performance on a metric we all care about. Apple has a different approach (own the entire stack) but they're going for the same thing and it has made them a lot of cash.
- Tloewald 10y ago
  Netflix is a rather specific example — don't they use Silverlight (as in "the only site anyone in the world cares about that uses Silverlight"?) It's like Apple stressing how well their browser works on -- oh wait, Apple doesn't produce a proprietary plugin. Well, let's say the iTunes page, if it were specifically tuned to work well in Safari.
  - ascagnel_ 10y ago
    Edge and Safari run Netflix under HTML5's protected media path. Silverlight is only used for backwards compatibility.
deleted 10y ago
[deleted]
ntoshev 10y ago
It may not be fair to enable adblocking if you want to compare other engineering components, but these results still mean users in the real world will see more battery time with Opera than with Edge.
JaRail 10y ago
This is * not based on observed user behavior * not reproducible by you (recorded/replayed network where possible) * not reproducible by peers * in stark contrast to reproducible tests provided by others You describe it as "highly scientific" and jump straight to a marketing campaign. Honestly, who's skipping a lap of the race? It's pretty clear the data generated is useful from an engineering point of view. It can help identify problem areas. It's also absolutely clear that the conclusions being marketed are blatant misrepresentations of the work.

wernerb 10y ago

The way I read the blog is that the adblocking is enabled to show that even _with_ an expensive extension enabled it still beats Edge, but that was in my opinion not the gist of the post. The post was about that microsoft is not transparent about the methodology/setup/scripts/target websites used. A third party should be able to support Microsoft's claims. Is it possible for the setup to be published to your github.com/microsoft so that it may be executed automatically? Heck, go the extra mile put it in a CI and publish the data on regular basis :)

IshKebab 10y ago
Adblock isn't an "expensive extension". Ads are the expensive thing, so blocking them saves power.
- wernerb 10y ago
  Until we know exactly what websites/videos etc are loaded (also on the Opera benchmark!); you, me and all consumers do not know if this is true :)
- DominikD 10y ago
  Also Opera has built-in blocking. It's not a JavaScript-heavy extension to begin with. ;)
- oxymoron 10y ago
  It's not quite that simple [0][1], or at least it wasn't always. AdBlock Plus in particular used to rely on a massive style sheet that was injected into all tabs and frames, which degraded really quickly on pages with a lot of iframes (which, as it happens, is typically pages with lots of ads). I believe Firefox partly resolved the problem on their end later on, although I can't tell you the exact status of things. Nevertheless, there's definitely some precedence for claiming that ad blocking is an expensive operation. Intuitively, you'd think that network based blocking would be enough, but it won't work against same domain ad sources (a typical example being facebook ads), while css selectors are able to capture a bit more depth. Nevertheless, I don't know exactly what has happened since, although I do recall Firefox making some adjustments on their end that improved the situation. [0] http://www.extremetech.com/computing/182428-ironic-iframes-adblock-plus-is-probably-the-reason-firefox-and-chrome-are-such-memory-hogs http://www.extremetech.com/computing/182428-ironic-iframes-a... [1] https://blog.mozilla.org/nnethercote/2014/05/14/adblock-pluss-effect-on-firefoxs-memory-usage/ https://blog.mozilla.org/nnethercote/2014/05/14/adblock-plus...
  - ryuuchin 10y ago
    No, it really is that simple[1]. The only reason I can think of for someone not using uBlock Origin is because they've never heard of it. [1] https://github.com/gorhill/uBlock#performance https://github.com/gorhill/uBlock#performance

spankalee 10y ago

Doesn't the Netflix test really test GPU decoding vs software decoding? I'm not 100% sure on this, so any clarification welcome, but I thought that Netflix content contracts DRM requirements allowed GPU decoding via Playready, but not on some others like Widevine. Would the Netflix test still show such an advantage for Edge if there weren't any DRM?

j_h_s 10y ago
It seems to me that if you're running on the same device, the gpu should not be a variable.
- caconym_ 10y ago
  IIRC everything except Edge and Safari are forced to use Silverlight for Netflix streaming. I'm not exactly sure why this is, but it probably has something to do with the mess that is video codec licensing.
  - jsmith0295 10y ago
    It's for DRM purposes
  - tyleregeto 10y ago
    Netflix plays fine in Chrome (maybe Firefox too) using HTML5, it was opt-in at one point not sure about now. I watch in Linux all the time, no Silverlight there.
    - russjr08 10y ago
      Starting with Firefox 47, it works fine as well (macOS here), but you need to switch your user agent to Chrome (before you go to Netflix). I'm guessing Netflix hasn't updated their compatibility checker.
  - ryuuchin 10y ago
    Chrome uses widevine (DRM) for Netflix which uses HTML5. Really the more significant difference is that Chrome gets capped to 720p. Edge actually gets the 1080p streams (as does IE11 on Windows 8+ and the Netflix Windows Store app).

8_hours_ago 10y ago

It would be interesting to see this test run again with ad blockers enabled on each browser (the latest version of Edge has support for extensions).

hristov 10y ago

So where is your script? Or are we just supposed to trust you?

kayone 10y ago
you don't have to do anything.

mynewtb 10y ago

How did you control for the specific ads the browsers were served? Ads are highly dynamic. Unless you (can) include the differences in your methodology you better eliminate that influence.

patcheudor 10y ago
Ding, ding, ding and to be fair this is the problem in both methodologies. The only way to accurately perform this test is to spider a bunch of sites, save the contents to a locally hosted HTTPD and ensure all third-party JS calls are resolved locally and test both against the exact same sites as they were spidered at a point in time. You can simply not account for changes which may happen to the markup, ad network beacons, ads, metrics code, or even network routing, all of which could influence the test in one way or the other if not run from an identical static cache in a controlled environment. If I were doing this test I'd then skip the whole scripting automation part and simply add a meta-refresh to every page in the cache to sequentially take the browser through the content, giving each page something like 10 seconds to load and render. Simple, simple, and far more accurate.
- csl 10y ago
  You don't need to go to such complicated lengths. Just perform enough tests (as in, a statistically large enough amount) and a distribution will form. That also captures the variability of real world network effects.
  - patcheudor 10y ago
    It's hardly complicated. I've put such tests together in an afternoon. In fact, whatever is added in complexity is gained by the fact fewer tests are necessary. Via this mechanism you can also remove any questions about compression, use of HTTP/2, etc., which could impact the tests based on server-side choices when it comes to serving data to either platform. Equal always equals better.
    - dogma1138 10y ago

caconym_ 10y ago

Given that basically everyone should be blocking ads (if for no reason other than that they are a vector for malware), native ad blocking is really an energy saving mode, a user experience improvement, and a crucial security measure all rolled up into one feature. If the focus of this test is meant to be on the user experience, as the tone of the Microsoft blog post seems to suggest, then said native ad blocker is a killer feature that really might mean Opera users can "stay productive longer". If not, then sure, Edge's underlying implementation is probably faster. It would be interesting to see a comparable benchmark of Edge with the best available ad blocking turned on.

hristov 10y ago

Wow this is all fascinating! If it is not too much trouble, would you mind releasing the script that you used in your testing so that independent parties may verify your results.

wdr1 10y ago

Could you disclose/open source your methodology?

Aissen 10y ago

> nobody seems to challenge our Netflix results: Of course, this is Lotus vs Office all over again. You're using the OS advantage (Windows 10 and its DRM platform) to show that you can do things more efficiently. You chose the technology platform (I'm guessing Play Ready), you forced GPU vendors to comply, and now you give competitors a bad reputation for daring to use their own (or a competitor's) technology.

rkangel 10y ago
Or, to put it another way: the have built a well integrated technology stack and are now demonstrating that it has superior performance on a metric we all care about. Apple has a different approach (own the entire stack) but they're going for the same thing and it has made them a lot of cash.
Tloewald 10y ago
Netflix is a rather specific example — don't they use Silverlight (as in "the only site anyone in the world cares about that uses Silverlight"?) It's like Apple stressing how well their browser works on -- oh wait, Apple doesn't produce a proprietary plugin. Well, let's say the iTunes page, if it were specifically tuned to work well in Safari.
- ascagnel_ 10y ago
  Edge and Safari run Netflix under HTML5's protected media path. Silverlight is only used for backwards compatibility.

deleted 10y ago

[deleted]

ntoshev 10y ago

It may not be fair to enable adblocking if you want to compare other engineering components, but these results still mean users in the real world will see more battery time with Opera than with Edge.

JaRail 10y ago

This is * not based on observed user behavior * not reproducible by you (recorded/replayed network where possible) * not reproducible by peers * in stark contrast to reproducible tests provided by others You describe it as "highly scientific" and jump straight to a marketing campaign. Honestly, who's skipping a lap of the race? It's pretty clear the data generated is useful from an engineering point of view. It can help identify problem areas. It's also absolutely clear that the conclusions being marketed are blatant misrepresentations of the work.