5 ms·

Web Request and Declarative Net Request: Explaining the impact on Extensions

52 points by migueldemoura 7y ago

SquareWheel 7y ago
They've addressed a number of complaints about limitations of the new API: >Since the original announcement of the Declarative Net Request API, we have added significant functionality to the API as a result of these discussions. The Declarative Net Request API now allows for the registration and removal of dynamic rules - specified at runtime rather than statically in the manifest. We’ve also added the capability to remove common tracking headers, such as Referer, Cookie, and Set-Cookie. >We are actively exploring other ways to expand this API, including adding methods to get feedback about matched rules, and support for richer redirects leveraging URL manipulation and regular expressions. Additionally, we are currently planning to change the rule limit from maximum of 30k rules per extension to a global maximum of 150k rules. They posted a simpler post to the Security blog also addressing these issues. https://security.googleblog.com/2019/06/improving-security-and-privacy-for.html https://security.googleblog.com/2019/06/improving-security-a... They lead with the point that most-seemed to get lost in translation, even on technical sites like HN. >No, Chrome isn’t killing ad blockers
- pilif 7y ago
  >They lead with the point that most-seemed to get lost in translation, even on technical sites like HN. >>No, Chrome isn’t killing ad blockers They’re just neutering them with a ridiculously low limit of rules. One of the benefits of doing this declaratively is that they can optimize the shit out of these blocking declarations. There is no need for such a low limit. Or even a limit at all.
  - SquareWheel 7y ago
    Even EasyList, which is probably the largest list out there, has 42,000 rules. So I wouldn't consider a 150,000 limit to be "ridiculously low". However I do agree that there shouldn't be a limit, in accordance with https://en.wikipedia.org/wiki/Zero_one_infinity_rule https://en.wikipedia.org/wiki/Zero_one_infinity_rule. I think they were just trying to get people to actually clean up EasyList to prune dead rules.
    - smileybarry 7y ago
      EasyList has 91k rules. Combine that with malware domain lists and other regional lists, and you can easily get to that limit. I just checked and my uBlock Origin -- which doesn't have any social filter lists applied -- has 148,339 rules applied in total.
      - SquareWheel 7y ago
        >EasyList has 91k rules. I read 42K from a source earlier today. That's where I got my figure. https://www.ghacks.net/2019/01/22/chrome-extension-manifest-v3-could-end-ublock-origin-for-chrome/ https://www.ghacks.net/2019/01/22/chrome-extension-manifest-... Might it be different for different extensions?
        danShumway 7y ago
        My current uBlock extension has 99,596 network filters. Easylist alone has 90,000 for me, the rest are miscellaneous ad-on lists -- so maybe it is different for different apps. The direct download from the official site has about 75,000 filters[0], though maybe that includes some cosmetic ones? 150,000 is definitely better than 30,000, but it's probably not a future-proof number. EDIT: Actually, given Apple's 50,000 filter limit, I wonder if your source was looking at a version of Easylist that was trimmed down specifically for Safari. [0]: https://easylist.to/easylist/easylist.txt https://easylist.to/easylist/easylist.txt
        Macha 7y ago
        I have 91k in ublock origin on my firefox install also.
- hlieberman 7y ago
  Killing? No. Handicapping? Yes. And why are we surprised? It’s in their best interest to do so; expecting a for-profit company to go against its own monetary best interests forever is naïve, at best.
  - SquareWheel 7y ago
    If you can block Google Ads equally as well in the new system as the old, how does this affect their "monetary best interests"?
  - swang 7y ago
    let's say they didn't up the limit from 30k. whose ads are getting blocked as a priority? doubleclick is going to be in there so is there really a difference to them either way.
  - dfabulich 7y ago
    The rule limits Google's proposing are more generous than the limits Apple puts on content blockers. Apple allows only 50K rules. And it turns out 50K rules is actually fine! Declarative rules engines are better and preserve privacy better. This is Google following Apple's lead and doing the right thing.
    - buckminster 7y ago
      It does nothing for privacy because extensions will still be able to see the requests, they just won't be able to block them.
      - dfabulich 7y ago
        No, manifest v3 extensions will only be able to observe requests for specific hosts, with permission from each host.
        buckminster 7y ago
        This isn't mentioned in the public version of the v3 spec (perhaps I'm looking in the wrong place) or in today's blog post. Do you have a reference?
snowwolf 7y ago
This seems like massive spin. Their primary argument doesn’t wash. As far as I understand it the web request API will still exist and still allow extension developers to view all request data. They just won’t be able to block the request and change it (I.e block content from loading)
- SquareWheel 7y ago
  As much as I've defended this change, I agree with you. The only justification I could see is that it might require an extra permission in an extension's manifest, but even that I'm not sure of. If this is part of a longer-term plan to deprecate webRequest and completely remove that ability, then they should say so now.
  - SquareWheel 7y ago
    This might create a confusing link loop, but a dev commented to explain this rationale now. https://twitter.com/justinschuh/status/1138889508512866304 https://twitter.com/justinschuh/status/1138889508512866304
    - snowwolf 7y ago
      So lets take an alternative look at this. The justification is that this improves privacy, security and performance. For who? I use 2 extensions, an Ad Blocker (uBlock Origin) and Password Manager. uBlock Origin has over 10M installs and the author has stated publicly that they cannot effectively do what they do if this change goes ahead. I can guarantee that uBlock Origin does more to protect my privacy, security and performance than this change. How many people have been affected by malicious extensions? I seriously doubt it's greater than 10M. So to improve the privacy, security and performance for a few, they are going to make it worse for the many. Surely there are better solutions to tackle this problem? Maintain the API but add in additional checks/vetting for any extensions that use it.
      - dfabulich 7y ago
deleted 7y ago
[deleted]
drenvuk 7y ago
No. It is incredibly hard to be cordial when you're trying to ruin the web and say that you're actually helping. I am livid. Because of content blockers the speed of web browsing has increased substantially. Because of content blockers users' security is vastly improved. ublock Origin has been doing the right thing and now you're going to make it harder. With the limits imposed by the declarative net requests at 150k developers and extensions that are safeguarding us are being hamstrung. In addition, because of the Chrome extension team's obsession with using js workers instead of persistent processes we're going to have to deal with finding work-arounds for holding important and unencrypted things in memory. Someone brought up the issue that some extensions will have to create a newly pinned tab to the top left of the browser because of these inane bullshit rules removing the background page. Imagine a stack of pinned tabs just so your extensions can do their jobs. Does that sound dumb enough yet? Sometimes it's necessary to hold a lot of information in memory because the user actually wants the extension to do a certain task in a reasonable amount of time. They are making our jobs harder because they're too stubborn to understand that extension developers are the ones who create the magic. This is central governance at its finest. These rules are stupid as hell.
- dfabulich 7y ago
  The rule limits Google's proposing are more generous than the limits Apple puts on content blockers. Apple allows only 50K rules. And it turns out 50K rules is actually fine! Declarative rules engines are better and preserve privacy better. This is Google following Apple's lead and doing the right thing.
  - buckminster 7y ago
    Please don't spam the same comment multiple times. https://news.ycombinator.com/item?id=20168111 https://news.ycombinator.com/item?id=20168111
  - mixedCase 7y ago
    > The rule limits Google's proposing are more generous than the limits Apple puts on content blockers And Firefox and most Chromium forks have no limit. Does that mean they have infinite generosity? Seriously, how is Apple being bad at something an argument for lowering other options to the same level? Right now I have ~200K filters enabled on this machine. EasyList alone is sitting at 90K according to uBlock Origin.
dang 7y ago
Related: https://news.ycombinator.com/item?id=20167246 https://news.ycombinator.com/item?id=20167246
danShumway 7y ago
> There’s been a lot of confusion and misconception around both the motivations and implications of this change Oh, heck off. This is Google's fallback whenever people call them out on anything -- "you just don't understand the entire picture." It's just their way of gaslighting critics. People understand the change perfectly. a) The privacy arguments are untrue because the Observational capabilities of the WebRequest API aren't being deprecated. There are no privacy benefits unless an extension voluntarily ignores that part of the API -- which no one will do because, as Google says: > when extension developers are given the choice between capability and security, the vast majority of developers choose capability. Google makes the argument that 42% of malicious extensions use this API. Well, guess what they'll still be able to do under Manifest V3? b) The vast majority of data suggests that ad blocker performance is not a serious issue on the web right now. Ad blockers pretty much universally help more than they hurt, so it's not clear why performance is such a big concern. c) The declarative model is a less powerful alternative, which is the exact reason why Google is deprecating the blocking capabilities in the first place -- if they didn't force it, no one would switch (again, see their "why not both?" argument). So what don't consumers and developers understand about this? It's an objectively less powerful API that does very little to improve privacy for the sake of performance improvements that aren't necessary.
- dfabulich 7y ago
  > The Observational capabilities of the WebRequest API aren't being deprecated They are being severely curtailed. https://twitter.com/justinschuh/status/1138889508512866304 https://twitter.com/justinschuh/status/1138889508512866304 > Yes, but network requests won't be observable without permission on the relevant hosts. And host permissions are changing significantly in Mv3 (e.g. they can't be granted as blanket install time permissions). > The underlying issue with blocking Web Request is that it is effectively a UXSS, which could not be bound by origin-based host permissions like the rest of the API can. Google's privacy argument is correct.
  - danShumway 7y ago
    I strongly disagree. There is pretty much no chance that every ad blocker and every other extension isn't going to widely request that new permission, because without it you can't get the data you need to maintain adblock lists or do more advanced detection (ie Privacy Badger). On the current discussion thread, Chrome devs say: > The webRequest API's ability to observe requests is foundational for extensions that modify their behavior based on the patterns they observe at runtime. And in this article, they say: > when extension developers are given the choice between capability and security, the vast majority of developers choose capability. So the argument that hiding observation requests behind a permission means that they are "severely curtailed" is just plain wrong. Most extensions (including ad blockers) will still request these permissions as wildcards across all hosts, and most users will still grant them as wildcards across all hosts. Per-domain permissions for the blocking WebRequest API would be great (that's a change I would broadly support), and if Chrome devs honestly believed that per-domain permissions were an adequate user protection against malware, they'd look for ways to implement per-domain permissions with the blocking API, rather than deprecating the entire thing. The Twitter post you link claims that's impossible, I don't see why. That would be a good thing for someone from Chrome to explain in either the massive developer thread that's been going on for months, or maybe in the article that complains about how much "confusion" users and developers have, instead of asking devs to blindly take their word.
ChrisCinelli 7y ago
> In addition to these safety concerns, there are also significant performance costs. In most cases, these costs are not from the evaluation of the extension script processing events, but rather from everything else coordinating the script. That overall performance impact can be very large, even for an extension written as performantly as possible where the JavaScript execution time is negligible. Cannot they just instrument the code that intercept the requests and tell the user when performances are affected? Something like "It seems that extension xyz is slowing down Chrome. Do you want to disable it? Yes No [] Do not ask anymore"

52 points by migueldemoura 7y ago

They've addressed a number of complaints about limitations of the new API: >Since the original announcement of the Declarative Net Request API, we have added significant functionality to the API as a result of these discussions. The Declarative Net Request API now allows for the registration and removal of dynamic rules - specified at runtime rather than statically in the manifest. We’ve also added the capability to remove common tracking headers, such as Referer, Cookie, and Set-Cookie. >We are actively exploring other ways to expand this API, including adding methods to get feedback about matched rules, and support for richer redirects leveraging URL manipulation and regular expressions. Additionally, we are currently planning to change the rule limit from maximum of 30k rules per extension to a global maximum of 150k rules. They posted a simpler post to the Security blog also addressing these issues. https://security.googleblog.com/2019/06/improving-security-and-privacy-for.html https://security.googleblog.com/2019/06/improving-security-a... They lead with the point that most-seemed to get lost in translation, even on technical sites like HN. >No, Chrome isn’t killing ad blockers

pilif 7y ago
>They lead with the point that most-seemed to get lost in translation, even on technical sites like HN. >>No, Chrome isn’t killing ad blockers They’re just neutering them with a ridiculously low limit of rules. One of the benefits of doing this declaratively is that they can optimize the shit out of these blocking declarations. There is no need for such a low limit. Or even a limit at all.
- SquareWheel 7y ago
  Even EasyList, which is probably the largest list out there, has 42,000 rules. So I wouldn't consider a 150,000 limit to be "ridiculously low". However I do agree that there shouldn't be a limit, in accordance with https://en.wikipedia.org/wiki/Zero_one_infinity_rule https://en.wikipedia.org/wiki/Zero_one_infinity_rule. I think they were just trying to get people to actually clean up EasyList to prune dead rules.
  - smileybarry 7y ago
    EasyList has 91k rules. Combine that with malware domain lists and other regional lists, and you can easily get to that limit. I just checked and my uBlock Origin -- which doesn't have any social filter lists applied -- has 148,339 rules applied in total.
    - SquareWheel 7y ago
      >EasyList has 91k rules. I read 42K from a source earlier today. That's where I got my figure. https://www.ghacks.net/2019/01/22/chrome-extension-manifest-v3-could-end-ublock-origin-for-chrome/ https://www.ghacks.net/2019/01/22/chrome-extension-manifest-... Might it be different for different extensions?
      - danShumway 7y ago
        My current uBlock extension has 99,596 network filters. Easylist alone has 90,000 for me, the rest are miscellaneous ad-on lists -- so maybe it is different for different apps. The direct download from the official site has about 75,000 filters[0], though maybe that includes some cosmetic ones? 150,000 is definitely better than 30,000, but it's probably not a future-proof number. EDIT: Actually, given Apple's 50,000 filter limit, I wonder if your source was looking at a version of Easylist that was trimmed down specifically for Safari. [0]: https://easylist.to/easylist/easylist.txt https://easylist.to/easylist/easylist.txt
      - Macha 7y ago
        I have 91k in ublock origin on my firefox install also.
hlieberman 7y ago
Killing? No. Handicapping? Yes. And why are we surprised? It’s in their best interest to do so; expecting a for-profit company to go against its own monetary best interests forever is naïve, at best.
- SquareWheel 7y ago
  If you can block Google Ads equally as well in the new system as the old, how does this affect their "monetary best interests"?
- swang 7y ago
  let's say they didn't up the limit from 30k. whose ads are getting blocked as a priority? doubleclick is going to be in there so is there really a difference to them either way.
- dfabulich 7y ago
  The rule limits Google's proposing are more generous than the limits Apple puts on content blockers. Apple allows only 50K rules. And it turns out 50K rules is actually fine! Declarative rules engines are better and preserve privacy better. This is Google following Apple's lead and doing the right thing.
  - buckminster 7y ago
    It does nothing for privacy because extensions will still be able to see the requests, they just won't be able to block them.
    - dfabulich 7y ago
      No, manifest v3 extensions will only be able to observe requests for specific hosts, with permission from each host.
      - buckminster 7y ago
        This isn't mentioned in the public version of the v3 spec (perhaps I'm looking in the wrong place) or in today's blog post. Do you have a reference?

snowwolf 7y ago

This seems like massive spin. Their primary argument doesn’t wash. As far as I understand it the web request API will still exist and still allow extension developers to view all request data. They just won’t be able to block the request and change it (I.e block content from loading)

SquareWheel 7y ago
As much as I've defended this change, I agree with you. The only justification I could see is that it might require an extra permission in an extension's manifest, but even that I'm not sure of. If this is part of a longer-term plan to deprecate webRequest and completely remove that ability, then they should say so now.
- SquareWheel 7y ago
  This might create a confusing link loop, but a dev commented to explain this rationale now. https://twitter.com/justinschuh/status/1138889508512866304 https://twitter.com/justinschuh/status/1138889508512866304
  - snowwolf 7y ago
    So lets take an alternative look at this. The justification is that this improves privacy, security and performance. For who? I use 2 extensions, an Ad Blocker (uBlock Origin) and Password Manager. uBlock Origin has over 10M installs and the author has stated publicly that they cannot effectively do what they do if this change goes ahead. I can guarantee that uBlock Origin does more to protect my privacy, security and performance than this change. How many people have been affected by malicious extensions? I seriously doubt it's greater than 10M. So to improve the privacy, security and performance for a few, they are going to make it worse for the many. Surely there are better solutions to tackle this problem? Maintain the API but add in additional checks/vetting for any extensions that use it.
    - dfabulich 7y ago

deleted 7y ago

[deleted]

drenvuk 7y ago

No. It is incredibly hard to be cordial when you're trying to ruin the web and say that you're actually helping. I am livid. Because of content blockers the speed of web browsing has increased substantially. Because of content blockers users' security is vastly improved. ublock Origin has been doing the right thing and now you're going to make it harder. With the limits imposed by the declarative net requests at 150k developers and extensions that are safeguarding us are being hamstrung. In addition, because of the Chrome extension team's obsession with using js workers instead of persistent processes we're going to have to deal with finding work-arounds for holding important and unencrypted things in memory. Someone brought up the issue that some extensions will have to create a newly pinned tab to the top left of the browser because of these inane bullshit rules removing the background page. Imagine a stack of pinned tabs just so your extensions can do their jobs. Does that sound dumb enough yet? Sometimes it's necessary to hold a lot of information in memory because the user actually wants the extension to do a certain task in a reasonable amount of time. They are making our jobs harder because they're too stubborn to understand that extension developers are the ones who create the magic. This is central governance at its finest. These rules are stupid as hell.

dfabulich 7y ago
The rule limits Google's proposing are more generous than the limits Apple puts on content blockers. Apple allows only 50K rules. And it turns out 50K rules is actually fine! Declarative rules engines are better and preserve privacy better. This is Google following Apple's lead and doing the right thing.
- buckminster 7y ago
  Please don't spam the same comment multiple times. https://news.ycombinator.com/item?id=20168111 https://news.ycombinator.com/item?id=20168111
- mixedCase 7y ago
  > The rule limits Google's proposing are more generous than the limits Apple puts on content blockers And Firefox and most Chromium forks have no limit. Does that mean they have infinite generosity? Seriously, how is Apple being bad at something an argument for lowering other options to the same level? Right now I have ~200K filters enabled on this machine. EasyList alone is sitting at 90K according to uBlock Origin.

dang 7y ago

Related: https://news.ycombinator.com/item?id=20167246 https://news.ycombinator.com/item?id=20167246

danShumway 7y ago

> There’s been a lot of confusion and misconception around both the motivations and implications of this change Oh, heck off. This is Google's fallback whenever people call them out on anything -- "you just don't understand the entire picture." It's just their way of gaslighting critics. People understand the change perfectly. a) The privacy arguments are untrue because the Observational capabilities of the WebRequest API aren't being deprecated. There are no privacy benefits unless an extension voluntarily ignores that part of the API -- which no one will do because, as Google says: > when extension developers are given the choice between capability and security, the vast majority of developers choose capability. Google makes the argument that 42% of malicious extensions use this API. Well, guess what they'll still be able to do under Manifest V3? b) The vast majority of data suggests that ad blocker performance is not a serious issue on the web right now. Ad blockers pretty much universally help more than they hurt, so it's not clear why performance is such a big concern. c) The declarative model is a less powerful alternative, which is the exact reason why Google is deprecating the blocking capabilities in the first place -- if they didn't force it, no one would switch (again, see their "why not both?" argument). So what don't consumers and developers understand about this? It's an objectively less powerful API that does very little to improve privacy for the sake of performance improvements that aren't necessary.

dfabulich 7y ago
> The Observational capabilities of the WebRequest API aren't being deprecated They are being severely curtailed. https://twitter.com/justinschuh/status/1138889508512866304 https://twitter.com/justinschuh/status/1138889508512866304 > Yes, but network requests won't be observable without permission on the relevant hosts. And host permissions are changing significantly in Mv3 (e.g. they can't be granted as blanket install time permissions). > The underlying issue with blocking Web Request is that it is effectively a UXSS, which could not be bound by origin-based host permissions like the rest of the API can. Google's privacy argument is correct.
- danShumway 7y ago
  I strongly disagree. There is pretty much no chance that every ad blocker and every other extension isn't going to widely request that new permission, because without it you can't get the data you need to maintain adblock lists or do more advanced detection (ie Privacy Badger). On the current discussion thread, Chrome devs say: > The webRequest API's ability to observe requests is foundational for extensions that modify their behavior based on the patterns they observe at runtime. And in this article, they say: > when extension developers are given the choice between capability and security, the vast majority of developers choose capability. So the argument that hiding observation requests behind a permission means that they are "severely curtailed" is just plain wrong. Most extensions (including ad blockers) will still request these permissions as wildcards across all hosts, and most users will still grant them as wildcards across all hosts. Per-domain permissions for the blocking WebRequest API would be great (that's a change I would broadly support), and if Chrome devs honestly believed that per-domain permissions were an adequate user protection against malware, they'd look for ways to implement per-domain permissions with the blocking API, rather than deprecating the entire thing. The Twitter post you link claims that's impossible, I don't see why. That would be a good thing for someone from Chrome to explain in either the massive developer thread that's been going on for months, or maybe in the article that complains about how much "confusion" users and developers have, instead of asking devs to blindly take their word.

ChrisCinelli 7y ago

> In addition to these safety concerns, there are also significant performance costs. In most cases, these costs are not from the evaluation of the extension script processing events, but rather from everything else coordinating the script. That overall performance impact can be very large, even for an extension written as performantly as possible where the JavaScript execution time is negligible. Cannot they just instrument the code that intercept the requests and tell the user when performances are affected? Something like "It seems that extension xyz is slowing down Chrome. Do you want to disable it? Yes No [] Do not ask anymore"