16 ms·

Chromium cleans up its act and daily DNS root server queries drop by 60B

208 points by bdcravens 6y ago

clan 6y ago
The is a news story which "reports" on a blog entry. At least they linked the source: https://blog.apnic.net/2021/02/04/how-chromium-reduces-root-dns-traffic/ https://blog.apnic.net/2021/02/04/how-chromium-reduces-root-...
- lstamour 6y ago
  Or https://blog.verisign.com/domain-names/chromiums-reduction-of-root-dns-traffic/ https://blog.verisign.com/domain-names/chromiums-reduction-o...
basilgohar 6y ago
Can someone explain why this couldn't have been implemented purely client side and validating if what was entered was first even a valid hostname? Or were the DNS queries only run after they passed an initial check? I think this is one of the key problems that emerged by merging the search functionality into the location bar. I still now always enable the separate search bar for Firefox and avoid running searches in the so-called Awesome bar. I still consider a search and host lookup very different operations in my mind.
- dublinben 6y ago
  Normal users don't enter hostnames directly any longer. Even for the most common sites like YouTube.com, Facebook.com and Amazon.com they will largely search the name and then click on the top search result. This behaviour actually suits Google well, because their incredibly lucrative "promoted results" are shown before the real destination site. You can't charge an advertising tax on users who browse directly to their destination.
  - oconnor663 6y ago
    Normal users don't type "fa" in the browser bar and get autocompleted to "facebook.com"?
- Deathmax 6y ago
  I believe it was mainly to detect when a user is trying to navigate to an intranet site with a single-word name (for example http://fileserver http://fileserver). To do that, Chrome needs to make a DNS request for the word and check if it's valid. However, you can have misbehaving ISPs that replaces all NXDOMAIN responses with something (producing false positives for what constitutes "valid" hostnames), or a guest portal that is hijacking DNS responses, so Chrome will make requests for random DNS hostnames that are unlikely to exist to detect if DNS hijacking is occurring and disable the intranet detection if it is.
  - lkbm 6y ago
    The APNIC post[0] backs this up, saying it's the "Intranet Redirect Detector". [1] has more details on how that system works (which is basically what you said, but slightly more in-depth). [0] https://blog.apnic.net/2021/02/04/how-chromium-reduces-root-dns-traffic/ https://blog.apnic.net/2021/02/04/how-chromium-reduces-root-... [1] https://www.archyde.com/chromium-function-against-dns-hijacking-puts-high-load-on-root-servers-computer-news/ https://www.archyde.com/chromium-function-against-dns-hijack...
  - somehnguy 6y ago
    If that was the goal, it does a pretty poor job. The amount of times I've accidentally Googled things like 'internalserver', 'fileserver:80', or 'dockerhost:8000' over the years is way too high. And when you do eventually reach any of them via manually prepending 'http:// http://', they make the 'http:// http://' not visible in the bar. What confusing signaling! Hint for Chrome: if I'm appending a port and the DNS name exists I probably don't want to Google it.
    - NicoJuicy 6y ago
      Counterpoint, i do want to Google ml.net
      - Moru 6y ago
        Then you type !g ml.net
- shakna 6y ago
  Worth noting that this doesn't actually affect the behaviour of a single user at this point in time. > Thanks, Matt, that's great to see. Note that that's the effect of the change in comment 37, which only affects Android. The desktop change is blocked on a planned experiment in M88. A wide rollout probably won't happen until late February or early March, at which point we expect an additional reduction in root traffic. [0] They were doing the DNS interception check on Android, but a different path of code was actually doing the checks (in a different way). Android was basically sending out useless requests. [0] https://bugs.chromium.org/p/chromium/issues/detail?id=1090985#c43 https://bugs.chromium.org/p/chromium/issues/detail?id=109098...
  - treesknees 6y ago
    Which amazes me. Chrome does this check every time network settings change, including a new IP. So your Android phone connecting to various networks just spammed this stuff for no reason. The fact that this fixed dropped root server requests by nearly 50% is astounding.
- rasengan 6y ago
  Handshake [1] handles validation on the client side before making a query. [1] https://handshake.org https://handshake.org
  - judge2020 6y ago
    Can you point to the specific piece of client side code that does this? It doesn’t sound like it inherently fixes the problem of either going to “myserver” (as in, http://myserver/ http://myserver/) and “myserver” (searching the web).
    - rasengan 6y ago
      Not all browsers are properly handling but placing a “.” after the name is a temporary fix.
CivBase 6y ago
I'm surprised 60 billion DNS queries is only 41% of the daily norm. With a billion people between the US and Europe alone - the vast majority of which use the world wide web directly multiple times a day (nevermind other online services) - and the amount of domains a typical website loads resources from, I figured daily DNS queries would have broken into the trillions by now. I suppose that's probably thanks to caching, dedicated apps for many websites, and most users sticking to a relatively small selection of websites.
- ElFitz 6y ago
  Also, people usually hit their ISPs’, Google’s or Cloudflare’s DNS servers, not the root ones It’s, I believe, those servers that hit the root DNS servers when they don’t have the data.
- jasonhansel 6y ago
  These are requests to the root servers, not to the ordinary DNS servers most people use.
  - sllabres 6y ago
    Correct, and IIRC the TTL is 5 days.
- csunbird 6y ago
  There is a lot of caching. https://xkcd.com/908/ https://xkcd.com/908/
- jlgaddis 6y ago
  Well, probably (at least) half of those billion people are using one of the large public DNS servers (Google, Cloudflare, et al.). Plus, the answers returned by the root servers more often than not include resource records with very high TTL values (e.g., NS RRs with a TTL of two days). These then get cached for that long by the recursive resolvers that are used (directly) by end users. The root servers aren't responding to requests for the A RR for google.com from Joe Schmoe or the MX RR for gmail.com from Outlook running on his desktop -- both of which (without checking) likely have TTLs measured in a two- or (at the most) three-digit number of seconds.
fitblipper 6y ago
At first blush the fact that all these searches were sent over DNS (which is plain text by default) this was a gross privacy violation. Congress has passed bills allowing ISPs to use your DNS and browsing history and sell it to 3rd parties [1]. With all these searchs being included in DNS queries chrome was basically allowing everyone who wanted that data that you were worried about that mole or that you have that weird fetish. [1] https://www.nbcnews.com/news/us-news/senate-votes-let-isps-sell-your-data-without-consent-n737921 https://www.nbcnews.com/news/us-news/senate-votes-let-isps-s... (edit: explaining better what I felt was a privacy violation)
- tptacek 6y ago
  Your link does not say what you say it does, though I do believe ISPs monetize DNS queries, which is part of the reason for DoH.
  - deleted 6y ago
    [deleted]
tyingq 6y ago
Timely, I suppose, since there's talk of partitioning DNS cache in the browser...which will drive it back up.
cashewchoo 6y ago
I recently completed a "software pilgrimage", as I decided to call it in explaining to friends why I did it, of writing my own DNS recursive resolver. I decided to implement it purely from RFC 1034/1035, with no library code aside from TCP and UDP sockets and a JSON config parser, those kinds of things. No library help for anything relating to the RFC. It's part of a homelab-focused DNS solution that I plan to release under the AGPLv3 once I dogfood it for a while. Basically "DNS for impatient homelabbers/SMB's who don't want to have to learn how to write zonefiles and don't want some crazy enterprise omnibus thing either". Anyway, in the process of doing this and putting it in my DHCP config and seeing the traffic from two smart TVs of different brands, my RIPE atlas probe, various phones and laptops, smart thermostat, etc etc, I've noticed that there is a TON of "garbage" dns requests. Like, the chart [the article] shows - where about 70% of queries result in a name error - that totally meshes with what I see on a much smaller scale. Right now, prometheus tells me that since last restart (which was about a week ago at this point?), I've answered 80437 queries with NO_ERROR, 52014 with NAME_ERROR, and 242 with SERVER_FAILURE (funnily enough, when spot checking these, 8.8.8.8 also SERVER_FAILUREs these same requests - usually devices with presumably-buggy dns libraries not correctly specifying lengths of variable-length fields). It really surprised my initial suspicions that so many DNS requests would be coming up with what I was originally considering to be an error condition. But I guess sometimes the absence of a DNS record is just as meaningful as the presence. Incidentally, I also noticed these chromium dns requests, and they had me worried for a bit because I wondered if they were malware trying to exploit some kind of vuln in dns servers. Took a bit of googling to figure it out. I do think they make up a decent % of the name errors I see, though I hadn't gotten around to having prometheus split them out to measure.
- fouronnes3 6y ago
  That's a very sweet idea. What are some other "software pilgrimage" projects for ambitious hackers with a finite amount of free time? Buying something in bitcoin using only the protocol spec and man pages? Tweeting from a Linux from scratch install? Writing a correct tar command on the first try? Writing a quine on your favorite language without google?
  - 6y ago
dpcx 6y ago
So is this functionality just "gone"? I remember one of the original reasons for this functionality was to know if you were on an ISP that performed any kind of DNS hijacking (I don't fully understand the why of it, but that's separate), and I'm curious if that's still available and if so, where are those queries actually going now?
- pas 6y ago
  Probably the long term solution is DNSSEC + DoH/DoT + trusted resolvers.

208 points by bdcravens 6y ago

clan 6y ago

The is a news story which "reports" on a blog entry. At least they linked the source: https://blog.apnic.net/2021/02/04/how-chromium-reduces-root-dns-traffic/ https://blog.apnic.net/2021/02/04/how-chromium-reduces-root-...

lstamour 6y ago
Or https://blog.verisign.com/domain-names/chromiums-reduction-of-root-dns-traffic/ https://blog.verisign.com/domain-names/chromiums-reduction-o...

basilgohar 6y ago

Can someone explain why this couldn't have been implemented purely client side and validating if what was entered was first even a valid hostname? Or were the DNS queries only run after they passed an initial check? I think this is one of the key problems that emerged by merging the search functionality into the location bar. I still now always enable the separate search bar for Firefox and avoid running searches in the so-called Awesome bar. I still consider a search and host lookup very different operations in my mind.

dublinben 6y ago
Normal users don't enter hostnames directly any longer. Even for the most common sites like YouTube.com, Facebook.com and Amazon.com they will largely search the name and then click on the top search result. This behaviour actually suits Google well, because their incredibly lucrative "promoted results" are shown before the real destination site. You can't charge an advertising tax on users who browse directly to their destination.
- oconnor663 6y ago
  Normal users don't type "fa" in the browser bar and get autocompleted to "facebook.com"?
Deathmax 6y ago
I believe it was mainly to detect when a user is trying to navigate to an intranet site with a single-word name (for example http://fileserver http://fileserver). To do that, Chrome needs to make a DNS request for the word and check if it's valid. However, you can have misbehaving ISPs that replaces all NXDOMAIN responses with something (producing false positives for what constitutes "valid" hostnames), or a guest portal that is hijacking DNS responses, so Chrome will make requests for random DNS hostnames that are unlikely to exist to detect if DNS hijacking is occurring and disable the intranet detection if it is.
- lkbm 6y ago
  The APNIC post[0] backs this up, saying it's the "Intranet Redirect Detector". [1] has more details on how that system works (which is basically what you said, but slightly more in-depth). [0] https://blog.apnic.net/2021/02/04/how-chromium-reduces-root-dns-traffic/ https://blog.apnic.net/2021/02/04/how-chromium-reduces-root-... [1] https://www.archyde.com/chromium-function-against-dns-hijacking-puts-high-load-on-root-servers-computer-news/ https://www.archyde.com/chromium-function-against-dns-hijack...
- somehnguy 6y ago
  If that was the goal, it does a pretty poor job. The amount of times I've accidentally Googled things like 'internalserver', 'fileserver:80', or 'dockerhost:8000' over the years is way too high. And when you do eventually reach any of them via manually prepending 'http:// http://', they make the 'http:// http://' not visible in the bar. What confusing signaling! Hint for Chrome: if I'm appending a port and the DNS name exists I probably don't want to Google it.
  - NicoJuicy 6y ago
    Counterpoint, i do want to Google ml.net
    - Moru 6y ago
      Then you type !g ml.net
shakna 6y ago
Worth noting that this doesn't actually affect the behaviour of a single user at this point in time. > Thanks, Matt, that's great to see. Note that that's the effect of the change in comment 37, which only affects Android. The desktop change is blocked on a planned experiment in M88. A wide rollout probably won't happen until late February or early March, at which point we expect an additional reduction in root traffic. [0] They were doing the DNS interception check on Android, but a different path of code was actually doing the checks (in a different way). Android was basically sending out useless requests. [0] https://bugs.chromium.org/p/chromium/issues/detail?id=1090985#c43 https://bugs.chromium.org/p/chromium/issues/detail?id=109098...
- treesknees 6y ago
  Which amazes me. Chrome does this check every time network settings change, including a new IP. So your Android phone connecting to various networks just spammed this stuff for no reason. The fact that this fixed dropped root server requests by nearly 50% is astounding.
rasengan 6y ago
Handshake [1] handles validation on the client side before making a query. [1] https://handshake.org https://handshake.org
- judge2020 6y ago
  Can you point to the specific piece of client side code that does this? It doesn’t sound like it inherently fixes the problem of either going to “myserver” (as in, http://myserver/ http://myserver/) and “myserver” (searching the web).
  - rasengan 6y ago
    Not all browsers are properly handling but placing a “.” after the name is a temporary fix.

CivBase 6y ago

I'm surprised 60 billion DNS queries is only 41% of the daily norm. With a billion people between the US and Europe alone - the vast majority of which use the world wide web directly multiple times a day (nevermind other online services) - and the amount of domains a typical website loads resources from, I figured daily DNS queries would have broken into the trillions by now. I suppose that's probably thanks to caching, dedicated apps for many websites, and most users sticking to a relatively small selection of websites.

ElFitz 6y ago
Also, people usually hit their ISPs’, Google’s or Cloudflare’s DNS servers, not the root ones It’s, I believe, those servers that hit the root DNS servers when they don’t have the data.
jasonhansel 6y ago
These are requests to the root servers, not to the ordinary DNS servers most people use.
- sllabres 6y ago
  Correct, and IIRC the TTL is 5 days.
csunbird 6y ago
There is a lot of caching. https://xkcd.com/908/ https://xkcd.com/908/
jlgaddis 6y ago
Well, probably (at least) half of those billion people are using one of the large public DNS servers (Google, Cloudflare, et al.). Plus, the answers returned by the root servers more often than not include resource records with very high TTL values (e.g., NS RRs with a TTL of two days). These then get cached for that long by the recursive resolvers that are used (directly) by end users. The root servers aren't responding to requests for the A RR for google.com from Joe Schmoe or the MX RR for gmail.com from Outlook running on his desktop -- both of which (without checking) likely have TTLs measured in a two- or (at the most) three-digit number of seconds.

fitblipper 6y ago

At first blush the fact that all these searches were sent over DNS (which is plain text by default) this was a gross privacy violation. Congress has passed bills allowing ISPs to use your DNS and browsing history and sell it to 3rd parties [1]. With all these searchs being included in DNS queries chrome was basically allowing everyone who wanted that data that you were worried about that mole or that you have that weird fetish. [1] https://www.nbcnews.com/news/us-news/senate-votes-let-isps-sell-your-data-without-consent-n737921 https://www.nbcnews.com/news/us-news/senate-votes-let-isps-s... (edit: explaining better what I felt was a privacy violation)

tptacek 6y ago
Your link does not say what you say it does, though I do believe ISPs monetize DNS queries, which is part of the reason for DoH.
- deleted 6y ago
  [deleted]

tyingq 6y ago

Timely, I suppose, since there's talk of partitioning DNS cache in the browser...which will drive it back up.

cashewchoo 6y ago

I recently completed a "software pilgrimage", as I decided to call it in explaining to friends why I did it, of writing my own DNS recursive resolver. I decided to implement it purely from RFC 1034/1035, with no library code aside from TCP and UDP sockets and a JSON config parser, those kinds of things. No library help for anything relating to the RFC. It's part of a homelab-focused DNS solution that I plan to release under the AGPLv3 once I dogfood it for a while. Basically "DNS for impatient homelabbers/SMB's who don't want to have to learn how to write zonefiles and don't want some crazy enterprise omnibus thing either". Anyway, in the process of doing this and putting it in my DHCP config and seeing the traffic from two smart TVs of different brands, my RIPE atlas probe, various phones and laptops, smart thermostat, etc etc, I've noticed that there is a TON of "garbage" dns requests. Like, the chart [the article] shows - where about 70% of queries result in a name error - that totally meshes with what I see on a much smaller scale. Right now, prometheus tells me that since last restart (which was about a week ago at this point?), I've answered 80437 queries with NO_ERROR, 52014 with NAME_ERROR, and 242 with SERVER_FAILURE (funnily enough, when spot checking these, 8.8.8.8 also SERVER_FAILUREs these same requests - usually devices with presumably-buggy dns libraries not correctly specifying lengths of variable-length fields). It really surprised my initial suspicions that so many DNS requests would be coming up with what I was originally considering to be an error condition. But I guess sometimes the absence of a DNS record is just as meaningful as the presence. Incidentally, I also noticed these chromium dns requests, and they had me worried for a bit because I wondered if they were malware trying to exploit some kind of vuln in dns servers. Took a bit of googling to figure it out. I do think they make up a decent % of the name errors I see, though I hadn't gotten around to having prometheus split them out to measure.

fouronnes3 6y ago
That's a very sweet idea. What are some other "software pilgrimage" projects for ambitious hackers with a finite amount of free time? Buying something in bitcoin using only the protocol spec and man pages? Tweeting from a Linux from scratch install? Writing a correct tar command on the first try? Writing a quine on your favorite language without google?
- 6y ago

dpcx 6y ago

So is this functionality just "gone"? I remember one of the original reasons for this functionality was to know if you were on an ISP that performed any kind of DNS hijacking (I don't fully understand the why of it, but that's separate), and I'm curious if that's still available and if so, where are those queries actually going now?

pas 6y ago
Probably the long term solution is DNSSEC + DoH/DoT + trusted resolvers.