8 ms·

Chromium's impact on root DNS traffic (2020)

101 points by jesprenj 4y ago

pigbearpig 4y ago
What an arrogant hack. It's hard to believe Google engineers would find that a reasonable approach. If I'd have seen that during a code review, I would have called it out and explained that you shouldn't abuse someone else's systems. Then again, maybe it's revenge for everyone pinging 8.8.8.8
- deleted 4y ago
  [deleted]
  - deleted 4y ago
    [deleted]
- mschuster91 4y ago
  The problem is, there is no better approach to check for intercepting middleboxes. Using a well-known path (like gstatic.com/generate_204) works for detecting if the user has a captive portal between them and the Internet, but not if the user's provider messes around with DNS.
  - blueflow 4y ago
    Google could host their own DNS servers who only give out NXD and test against that.
    - mschuster91 4y ago
      As soon as Google would do that, providers would implement exceptions.
      - adql 4y ago
        They could do DPI, allow traffic for unknown domains, intercept everything else. Differentiating random domain from "what user might actually type by accident" is also pretty trivial
  - inetknght 4y ago
    > there is no better approach to check for intercepting middleboxes I can see some purposes for detecting middleboxes. I've done it. It usually doesn't involve DNS. It does involve certificate pinning though. > detecting if the user has a captive portal between them and the Internet That's easy. Try to browse to something. If succeeds but the certificate isn't valid then the user probably has a captive portal. That, or your pinned certificate has been revoked. > detecting ... if the user's provider messes around with DNS Certificate pinning, again, comes to the rescue. Pin a certificate to your own DoH server and then use DoH to look up whatever you need. If you can't connect to your DoH server then you effectively aren't (or shouldn't be) connected to the internet.
    - opheliate 4y ago
      The kind of interception that is being discussed here is the interception of DNS queries which would return an NXDOMAIN result to instead point to an advertising page. This is done by a lot of ISPs.
      - sumtechguy 4y ago
        It is an interesting tricky problem. As the DNS system was designed for caching and an 'honor system'. How do you allow caching so you do not completely overwhelm the root servers yet still prevent modification/interception of the contents of the messages? DoH sort of addresses it. But then you still have to trust that the message has not been altered/intercepted at the DoH server. Yet we also want the server to be able to do a replay of that message to another client. Also how to retrofit all of the existing code that uses DNS that uses this magic system. It is like we need a lightweight one way TLS session but with certificate caching and a second request/response type with a hash. But you have to watch out for cert forging then. That the browsers are reacting in this way says we have a failure at the DNS software level. Those projects do seem to be giving the security that is wanted. So we are starting to get some fragmentation. Which can not be good. Perhaps we need new record types to support this?
        tptacek 4y ago
  - adql 4y ago
    the middlebox could just pass traffic for the unknown TLDs tho and that detectin method would be in vain
- a13o 4y ago
  Google makes money when people use 8.8.8.8, so there's an even more sinister interpretation. Chrome has all it's users DDoS other DNS servers, to raise the table stakes on running one. Now your infrastructure needs to handle apparently a whole second internet of bogus traffic. Google, of course, has the resources to meet this higher bar.
- plextoria 4y ago
  Well, Google isn't the only abuser here. The networks that hijack DNS request should share some of the blame, no?
  - dspillett 4y ago
    Chromium is imposing the extra load on the system, so yes it is the abuser in that sense. That they are doing it for what is a good reason for the app's users is immaterial to whether the effect is bad for the root DNS servers. If Karen takes my sandwich from the company fridge, so I take some of Jon's lunch, so I don't starve, I'm not innocent because Karen started it, I've created a situation where there are two arseholes instead of one. This isn't quite what is happening here as the root servers are effectively a public resource and stuff in the fridge is all private resources, but close enough to make the point. > The networks that hijack DNS request should share some of the blame They should have all the blame for deliberately breaking part of agreed protocols for their own gain. But that doesn't make anything we do in response to that right by virtue of us doing it because we have been wronged.
  - tptacek 4y ago
    This isn't an "abuse". These are the Internet root DNS servers. We don't design applications to tiptoe around them any more than we tiptoe around the capacity of core routers. These is a huge amount of capacity and all of root DNS together is a rounding error on total Internet traffic.
  - jeroenhd 4y ago
    They should, but Google's "solution" to the hijacking is the problem here. If you're going to hijack NXDOMAINs, you can just ignore requests to non-existent TLDs in your scheme and Chrome will be none the wiser. Google manages entire TLDs, surely they can use their own DNS servers for this purpose.
- capableweb 4y ago
  > What an arrogant hack. It's hard to believe Google engineers would find that a reasonable approach. Reminds me of the recent "Go module mirror fiasco" where Google found it fair to clone repositories at a rate of ~2,500 per hour in order to essentially proxy Go modules. - "Sourcehut will blacklist the Go module mirror" - https://news.ycombinator.com/item?id=34310674 https://news.ycombinator.com/item?id=34310674 After the drama become very much public, they finally decided to address the issue in a good way.
  - tptacek 4y ago
    That doesn't at all seem like what happened (the changes were in the works prior to the drama), but this is a high-drama tangent unrelated to the story here. This isn't some tiny authoritative DNS server being flooded unexpectedly with queries. These are the Internet DNS root services. They have to keep up with this kind of traffic. It's their literal job description.
  - adql 4y ago
    To be fair, that's less than one a second. But I dislike the whole GOPROXY design, like it breaking private repos by default and having to set some env variables to make this stupid tool download stuff from server I told it to download.
    - capableweb 4y ago
      > To be fair, that's less than one a second. Well, taking the context into consideration, I'd still say it's too much. Context being they were full git clones and the traffic ended up representing "70% of all outgoing network traffic from git.sr.ht".
  - deleted 4y ago
    [deleted]
- oaiey 4y ago
  Well .. the root servers and DNS platform in general provide name resolution. Traditionally we think that as a feature/function name -> IP. But there non functional aspects also to consider. Distribution is one (DNS is really good on that), security (DNSSEC comes to mind), multi-transport channel (e.g. DNS over HTTP) are others but like seen in this case there is a non functional need, that interception can be detected. DNS does not deliver that (yet). But it is clearly a need of it in the DNS platform. So calling it abuse, is wrong. It is a dirty hack for a non-existing feature. It is technical debt of the DNS platform and the root server suffer for it because the ISPs and in-house DNS resolvers create the problem. Being angry at Google we can anyway be. They have enough money, enough people and enough power to either fix this financially or as a feature within the DNS platform.
- deleted 4y ago
  [deleted]
- BenjiWiebe 4y ago
  I've switched to 'ping 1.1' (1.0.0.1).
mschuster91 4y ago
Once again a typical tragedy of the commons problem: some actors (in this case, providers) became too greedy, fucked up service for people, and because of the needed defense against such practices now the commonwealth (in this case, the operators of the root DNS servers) suffers. On top of that come captive hotspot/corporate portals - and here especially, I do wonder why messing around with DPI is necessary, when every public network has a DHCP server running that could be used to distribute DHCP options for gateway URLs or information if the connection is to be considered metered (mobile phone hotspot) or bandwidth constrained (train or bus hotspots).
- jeroenhd 4y ago
  If you're going for network login pages, you can intercept any of the standard HTTP URLs. There are specific URLs by Apple, Google, Microsoft, Mozilla, and if you're willing, a bunch of Linux distros as well, that solely serve to detect MitM redirects and show login prompts. DHCP isn't reliable as many clients don't do anything with advanced settings you provide. Good luck getting a phone to accept the proxy server you've configured over DHCP. This legitimate-ish DPI usually runs at the network itself, it doesn't traverse the uplink to cause any load.
  - tptacek 4y ago
    This isn't a cat-and-mouse game between DPIs and the browser; it's a hamfisted ad-tech tactic deployed by ISPs that straight-up breaks a browser feature, and the browser using the DNS to unbreak it.
jeffbee 4y ago
This seems to add < 1gbps globally across all roots? Seems minor?
- _qua 4y ago
  Am I misreading the article or does it show that this accounts for 50% of the dns traffic the root servers see? I don’t think that’s minor.
  - tptacek 4y ago
    It's the most popular implementation of what is by a very wide margin the most popular and important app on the Internet. What percentage of traffic would you have assumed it represented?
    - debugnik 4y ago
      Less than genuine lookups? Even if all the world's DNS traffic came from Chrome, probe lookups are taking 50% of all root server traffic just to confirm NXDOMAIN over and over.
      - tptacek 4y ago
        The browser confirms it at startup and when DNS settings change, not "over and over"; the "over and over" comes from the fact that zillions of people use Chrome. Serving the zillions of people who actually use the Internet is the job of the root servers. If they wanted to serve a small number of requests, they could go be the authority server for zombo.com instead.
        deleted 4y ago
        [deleted]
    - dspillett 4y ago
      A lot less than that. The point of the hierarchy of DNS caching is to keep traffic at the root (and points of authority) low. I would assume that the vast majority of traffic to/from root DNS servers to be due to access by caching resolver daemons not individual user applications, especially not one particular user application (no matter how popular).
Aachen 4y ago
Answering the implied question in the title: ~45% of traffic to root servers is now Chromium users figuring out whether nonexistent domains get correct nxdomain responses. That's apparently 60 billion queries per day. Doing the math, a query is some 92 bytes on the wire (50 bytes UDP payload, but the root servers are probably also on ethernet or something similar so I'll include headers), so 487 megabits per second per root server is taken up by this, assuming there are no spikes and everything is perfectly averaged. Edit: and that's the downlink. I forgot the uplink traffic that will be larger due to DNS' nature of echoing the query back with the response(s) attached, plus DNSSEC... Woa. /edit. So what's the next step, Google generously stepping up to running (some of) those root servers and gaining further control and observability over what people do online? Reading further, Firefox solves this by "namespace probe queries, directing them away from the root servers towards the browser’s infrastructure".
- deleted 4y ago
  [deleted]
- px43 4y ago
  Do those numbers look big to you? Looks to me like I could host a DNS root server in my garage with a normal consumer grade gigabit fiber connection. For infrastructure hosted in a legit data center, this is nothing.
  - jeffbee 4y ago
    Not only could you do so, if you did not care about reliability or latency you could host the entire DNS root on one desktop computer. Global root DNS traffic is nothing.
- enneff 4y ago
  Google doesn’t need to offer anything because the servers are holding up just fine under the load. These systems are trivially simple and cheap to operate by global internet standards.
lxgr 4y ago
Wow! Now I feel less bad about using "ping 8.8.8.8" as my canonical "is my internet uplink down" command…
- smcl 4y ago
  I think a couple of ICMP requests from a tiny subset of all internet users every now and then isn't too much strain, but if you want to lighten the load you can try alternating between Google's 8.8.8.8 and Cloudflare's 1.1.1.1 :)
- trishmapow2 4y ago
  CloudFlare is better since you can type ping 1.1 which becomes 1.0.0.1
  - lxgr 4y ago
    Yes, but how do I know that there isn't a Cloudflare edge node somewhere in my storage cabinet, announcing anycast routes for 1.0.0.1 and 1.1.1.1 to reduce latency?
tptacek 4y ago
In almost any other scenario, this traffic would be indistinguishable from a distributed denial of service (DDoS) attack. Yes, but see, it's not a distributed denial of service attack. It's a productive use of a capability the DNS roots were designed to serve. It's a huge amount of traffic because Chrome is one of the 3 most important and popular applications on the Internet, and the most popular of those, at that. The cart doesn't drag the horse.
- dpcx 4y ago
  I think "productive" depends on who's paying the bill and whether the uses of it were how the system was actually designed. I don't think that the roots ever had the idea of serving random requests billions of times because that's the entire point of having the downstream servers and DNS caching. I remember a time when directly pinging the root servers was considered very bad practice because you should always be using the downstream... Google's method essentially makes those downstream servers useless.
  - tptacek 4y ago
    All of these norms about DNS queries are deployed situationally. Namedroppers types angry about excess root queries today will defend the notion of end-user operating systems querying the roots directly to facilitate DNSSEC (which without direct recursive lookups from end-systems condenses down to a single "yes I did DNSSEC" bit set by your local DNS server). Everybody can look at this DNS probe code and kibitz about whether it could have been done more efficiently or whether it should be done at all, but this is a straightforward use of the DNS to do something important for end-users (detect whether their ISP's nameserver is lying to them about NXDOMAIN). Whatever else it is, it isn't a DDOS attack, and whatever else these servers are, they're not quiet out-of-the-way private systems; they're core Internet infrastructure.
    - jeffbee 4y ago
      People seem to think DNS root queries are answered by monks or something. There are ~1000 anycast sites serving the 13 roots and their load is close enough to zero to be ignored. Even if Chrome causes half of the traffic, who cares?
fwlr 4y ago
A comment on that post: “I’m the original author of this code, though I no longer maintain it. ”Just want to give folks a heads-up that we’ve been in discussion with various parties about this for some time now, as we agree the negative effects on the root servers are undesirable. So in terms of “do the Chromium folks know/care”; yes, and yes. ”This is a challenging problem to solve, as we’ve repeatedly seen in the past that NXDOMAIN hijacking is pervasive (I can’t speak quantitatively or claim that “exception rather than the norm” is false, but it’s certainly at least very common), and network operators often actively worked around previous attempts to detect it in order to ensure they could hijack without Chromium detecting it. ”We do have some proposed designs to fix; at this point much of the issue is engineering bandwidth. Though I’m sure that if ISPs wanted to cooperate with us on a reliable way to detect NXDOMAIN hijacking, we wouldn’t object. Ideally, we wouldn’t have to frustrate them, nor they us.” A way for DNS root servers and Chromium devs to cooperate that couldn’t be hijacked by domain redirecting ISPs would be nice.
- sneak 4y ago
  Chrome simply talking DoH to Google (or other trustworthy non-hijacking) servers by default would fix this totally.
  - tptacek 4y ago
    Fix what, though? This is an insignificant load on a piece of core infrastructure for which all of the load from all of the users of the world's most popular Internet application is still a rounding error error relative to capacity. This is what those servers are there to do!
    - jeroenhd 4y ago
      Those servers are for looking up real domains, they're not designed to be a detection mechanism for malicious actors. Chrome intentionally fires unresolveable uncacheable domains at the root servers. That's dishonest use of public infrastructure at the very least. I doubt Google would be happy if Microsoft or Apple were to build a network availability tool into their systems that executes random Google queries that won't get you a result to see if searches are being redirected.
NoZebra120vClip 4y ago
Something that doesn't seem to be directly addressed here is the data-leaking nature of this thing. I don't know what a typical user of Chromium expects to happen when they type arbitrary strings into the Omnibox, but I can tell you that they probably don't expect the full text to be forwarded to several random, centralized, and highly popular DNS servers that are neither under the control of Google nor their ISPs nor their employers/institutions. This may have the effect of logging all sorts of stuff that normally Google would be hoovering up, but instead it's in the hands of third parties. Extremely trustworthy parties indeed, but they are also high-value high-stakes targets for anyone who'd want to steal valuable data in the form of query logs. Obviously there's no good solution to this. I believe Firefox led the charge with the "Awesome Bar" or whatever it was first called, which habituated me and millions of users to type search queries directly into the URL bar. It's one of those features that's so amazingly useful that it really forces the giants to carefully calculate the tradeoff costs, such as the one discussed in TFA.
- m4lvin 4y ago
  Wait, I thought the check here is first done with random strings, and only if those return NXDOMAIN as expected, then the real search entries that might be hostnames are sent to your configured DNS server (and only after pressing enter).
  - tptacek 4y ago
    You're correct, that's what this post is about: random strings forwarded to the DNS roots to establish, without leaking any intent to the user's ISP DNS server, whether DNS NXDOMAINs are being intercepted. There's no privacy implication to this beyond the privacy benefit users get from things that push back on interception.
  - deleted 4y ago
    [deleted]
Thaxll 4y ago
60B queries a day is nothing, we're talking about root servers that serve as fundation for the internet as a whole. The other thing that people would be very suprised about is how old the software is on those root server, forget modern libraries with Rust/c++ and the like, it's pretty old tech that is very inefficient. Edit: when talking about old tech I'm talking about the architecture of the DNS server used, IO libraries and models, caching, data structure and the like, for example a lot of stuff has been done arround web servers to serve things very efficiently, the same could be done on DNS servers.
- nhooyr 4y ago
  > it's pretty old tech that is very inefficient. There is nothing about Rust or C++ that make them faster than C. In what way are the root servers inefficient?
- 7steps2much 4y ago
  > it's pretty old tech that is very inefficient. I can guarantee you that this "old tech" was coded with more thought invested into it than at least half of these "modern libraries". Well written C code can easily blow a C++/Rust application out of the water.
  - dralley 4y ago
    Provide some, any evidence whatsoever for this assertion.
    - 7steps2much 4y ago
      > Our measurement based on the micro benchmarks shows that Rust is in general slower than C, but the extent of the slowdown varies across different programs. On average, Rust brings a 1.77x “performance overhead” compared to C. [1] Of course, for the sake of completeness it should be noted that: > With the run-time checks disabled and the restrictions loosened, Rust presents a performance indistinguishable from C. [1] Though I believe my original statement to hold none the less, as disabling these restrictions disables (amongst other things) bound and overflow checking, which is one of if not the major selling point of rust. As for C++ depends on the features that one uses. If one writes "just what one could do in C" then the machine code produced by the compiler will be exactly (almost) the same. This is due to the fact that many c++ features are only compiler relevant but compile to (almost) the same instructions as code. However, I would once again raise the question I did above with rust: If we use little to no c++ features then can we distinguish that codebasse from a c codebase in any meaningful way? But assuming we write idomatic code we will have the c++ code behaving somewhat slower due to factors such as: - automatic collections/object allocation. Datastructures growing "on demand" do in general perform slower than a comparable "none automatic" datastructure increased in larger chunks by hand (using malloc/etc. in C). While this is an implementation detail admittedly, I believe the libstdc++ does not use chunking, though I would not swear on that. - strings. While no doubt a big upgrade from \0 terminated char sequences idomatic strings in C are less efficient. Especially when it comes to concatenating or manipulation of said strings. In addition it may lead to memory fragmentation, though this should be an afterthought most of the time. In general the performance difference of C++/C comes down to "hidden" code. While by no means large, assuming software such as the dns root servers which are running essentially 24/7 and will most likely continue doing so for quite a while even small differences in performance will add up. Admittedly however my original statement of > Well written C code can easily blow a C++/Rust application out of the water. May not have been well formulated. It would have been better to split the statement and be more specific about the individual performance differences in regards to rust/c++ instead of bunching them together. [1]: https://dl.acm.org/doi/pdf/10.1145/3551349.3559494 https://dl.acm.org/doi/pdf/10.1145/3551349.3559494
jandrese 4y ago
All of this extra traffic is to support the misfeature where the URL bar and search bar are combined! They are two different functions. When I type a domain I don't want the browser to send that straight to Google and give me a search result page. It makes a little bit of sense on phones where space is tight, but on desktop browsers it's just annoying.
- tptacek 4y ago
  It's an extraordinarily useful feature that millions of users take advantage of every day. The idea that the operators of core Internet infrastructure would be pressuring applications not to have these kinds of features is what should alarm you, not that the features exist. The cart doesn't drag the horse.
- leeoniya 4y ago
  > I don't want G wants it
- RockRobotRock 4y ago
  You probably use Firefox anyways, so why do you care? A lot of people like this feature.
  - jandrese 4y ago
    Sadly switching back to the old behavior has problems in Firefox. If you go to google.com and start typing in the search box it will redirect to the URL bar, where your search will fail because it is not a search box.
enneff 4y ago
All these indignant comments but nobody bothers to look up what happened since 2020: > In a previous blog post, we quantified that upwards of 45.80% of total DNS traffic to the root servers was, at the time, the result of Chromium intranet redirection detection tests. Since then, the Chromium team has redesigned its code to disable the redirection test on Android systems and introduced a multi-state DNS interception policy that supports disabling the redirection test for desktop browsers. This functionality was released mid-November of 2020 for Android systems in Chromium 87 and, quickly thereafter, the root servers experienced a rapid decline of DNS queries. https://blog.verisign.com/domain-names/chromiums-reduction-of-root-dns-traffic/ https://blog.verisign.com/domain-names/chromiums-reduction-o...
- pkasting 4y ago
  Indeed. Here's the bug where we fixed it: https://bugs.chromium.org/p/chromium/issues/detail?id=1090985 https://bugs.chromium.org/p/chromium/issues/detail?id=109098...
onphonenow 4y ago
Google would be happy to host the bandwidth for all DNS if needed. The idea that this bandwidth is relevant vs YouTube etc is crazy
tinus_hn 4y ago
Given that Chrome is the biggest browser by far, and apparently blocks these redirections, why do ISPs even bother with them?
nubinetwork 4y ago
Weird malware-like DNS queries aside, don't all browsers treat text in the URL bar that doesn't begin with http:// http:// or https:// https:// as search queries? Honestly, this is why I prefer using FQDN's, and bookmarks. I ask for what I want, and I get it (barring some captive portal, etc).

101 points by jesprenj 4y ago

What an arrogant hack. It's hard to believe Google engineers would find that a reasonable approach. If I'd have seen that during a code review, I would have called it out and explained that you shouldn't abuse someone else's systems. Then again, maybe it's revenge for everyone pinging 8.8.8.8

deleted 4y ago
[deleted]
- deleted 4y ago
  [deleted]
mschuster91 4y ago
The problem is, there is no better approach to check for intercepting middleboxes. Using a well-known path (like gstatic.com/generate_204) works for detecting if the user has a captive portal between them and the Internet, but not if the user's provider messes around with DNS.
- blueflow 4y ago
  Google could host their own DNS servers who only give out NXD and test against that.
  - mschuster91 4y ago
    As soon as Google would do that, providers would implement exceptions.
    - adql 4y ago
      They could do DPI, allow traffic for unknown domains, intercept everything else. Differentiating random domain from "what user might actually type by accident" is also pretty trivial
- inetknght 4y ago
  > there is no better approach to check for intercepting middleboxes I can see some purposes for detecting middleboxes. I've done it. It usually doesn't involve DNS. It does involve certificate pinning though. > detecting if the user has a captive portal between them and the Internet That's easy. Try to browse to something. If succeeds but the certificate isn't valid then the user probably has a captive portal. That, or your pinned certificate has been revoked. > detecting ... if the user's provider messes around with DNS Certificate pinning, again, comes to the rescue. Pin a certificate to your own DoH server and then use DoH to look up whatever you need. If you can't connect to your DoH server then you effectively aren't (or shouldn't be) connected to the internet.
  - opheliate 4y ago
    The kind of interception that is being discussed here is the interception of DNS queries which would return an NXDOMAIN result to instead point to an advertising page. This is done by a lot of ISPs.
    - sumtechguy 4y ago
      It is an interesting tricky problem. As the DNS system was designed for caching and an 'honor system'. How do you allow caching so you do not completely overwhelm the root servers yet still prevent modification/interception of the contents of the messages? DoH sort of addresses it. But then you still have to trust that the message has not been altered/intercepted at the DoH server. Yet we also want the server to be able to do a replay of that message to another client. Also how to retrofit all of the existing code that uses DNS that uses this magic system. It is like we need a lightweight one way TLS session but with certificate caching and a second request/response type with a hash. But you have to watch out for cert forging then. That the browsers are reacting in this way says we have a failure at the DNS software level. Those projects do seem to be giving the security that is wanted. So we are starting to get some fragmentation. Which can not be good. Perhaps we need new record types to support this?
      - tptacek 4y ago
- adql 4y ago
  the middlebox could just pass traffic for the unknown TLDs tho and that detectin method would be in vain
a13o 4y ago
Google makes money when people use 8.8.8.8, so there's an even more sinister interpretation. Chrome has all it's users DDoS other DNS servers, to raise the table stakes on running one. Now your infrastructure needs to handle apparently a whole second internet of bogus traffic. Google, of course, has the resources to meet this higher bar.
plextoria 4y ago
Well, Google isn't the only abuser here. The networks that hijack DNS request should share some of the blame, no?
- dspillett 4y ago
  Chromium is imposing the extra load on the system, so yes it is the abuser in that sense. That they are doing it for what is a good reason for the app's users is immaterial to whether the effect is bad for the root DNS servers. If Karen takes my sandwich from the company fridge, so I take some of Jon's lunch, so I don't starve, I'm not innocent because Karen started it, I've created a situation where there are two arseholes instead of one. This isn't quite what is happening here as the root servers are effectively a public resource and stuff in the fridge is all private resources, but close enough to make the point. > The networks that hijack DNS request should share some of the blame They should have all the blame for deliberately breaking part of agreed protocols for their own gain. But that doesn't make anything we do in response to that right by virtue of us doing it because we have been wronged.
- tptacek 4y ago
  This isn't an "abuse". These are the Internet root DNS servers. We don't design applications to tiptoe around them any more than we tiptoe around the capacity of core routers. These is a huge amount of capacity and all of root DNS together is a rounding error on total Internet traffic.
- jeroenhd 4y ago
  They should, but Google's "solution" to the hijacking is the problem here. If you're going to hijack NXDOMAINs, you can just ignore requests to non-existent TLDs in your scheme and Chrome will be none the wiser. Google manages entire TLDs, surely they can use their own DNS servers for this purpose.
capableweb 4y ago
> What an arrogant hack. It's hard to believe Google engineers would find that a reasonable approach. Reminds me of the recent "Go module mirror fiasco" where Google found it fair to clone repositories at a rate of ~2,500 per hour in order to essentially proxy Go modules. - "Sourcehut will blacklist the Go module mirror" - https://news.ycombinator.com/item?id=34310674 https://news.ycombinator.com/item?id=34310674 After the drama become very much public, they finally decided to address the issue in a good way.
- tptacek 4y ago
  That doesn't at all seem like what happened (the changes were in the works prior to the drama), but this is a high-drama tangent unrelated to the story here. This isn't some tiny authoritative DNS server being flooded unexpectedly with queries. These are the Internet DNS root services. They have to keep up with this kind of traffic. It's their literal job description.
- adql 4y ago
  To be fair, that's less than one a second. But I dislike the whole GOPROXY design, like it breaking private repos by default and having to set some env variables to make this stupid tool download stuff from server I told it to download.
  - capableweb 4y ago
    > To be fair, that's less than one a second. Well, taking the context into consideration, I'd still say it's too much. Context being they were full git clones and the traffic ended up representing "70% of all outgoing network traffic from git.sr.ht".
- deleted 4y ago
  [deleted]
oaiey 4y ago
Well .. the root servers and DNS platform in general provide name resolution. Traditionally we think that as a feature/function name -> IP. But there non functional aspects also to consider. Distribution is one (DNS is really good on that), security (DNSSEC comes to mind), multi-transport channel (e.g. DNS over HTTP) are others but like seen in this case there is a non functional need, that interception can be detected. DNS does not deliver that (yet). But it is clearly a need of it in the DNS platform. So calling it abuse, is wrong. It is a dirty hack for a non-existing feature. It is technical debt of the DNS platform and the root server suffer for it because the ISPs and in-house DNS resolvers create the problem. Being angry at Google we can anyway be. They have enough money, enough people and enough power to either fix this financially or as a feature within the DNS platform.
deleted 4y ago
[deleted]
BenjiWiebe 4y ago
I've switched to 'ping 1.1' (1.0.0.1).

mschuster91 4y ago

Once again a typical tragedy of the commons problem: some actors (in this case, providers) became too greedy, fucked up service for people, and because of the needed defense against such practices now the commonwealth (in this case, the operators of the root DNS servers) suffers. On top of that come captive hotspot/corporate portals - and here especially, I do wonder why messing around with DPI is necessary, when every public network has a DHCP server running that could be used to distribute DHCP options for gateway URLs or information if the connection is to be considered metered (mobile phone hotspot) or bandwidth constrained (train or bus hotspots).

jeroenhd 4y ago
If you're going for network login pages, you can intercept any of the standard HTTP URLs. There are specific URLs by Apple, Google, Microsoft, Mozilla, and if you're willing, a bunch of Linux distros as well, that solely serve to detect MitM redirects and show login prompts. DHCP isn't reliable as many clients don't do anything with advanced settings you provide. Good luck getting a phone to accept the proxy server you've configured over DHCP. This legitimate-ish DPI usually runs at the network itself, it doesn't traverse the uplink to cause any load.
- tptacek 4y ago
  This isn't a cat-and-mouse game between DPIs and the browser; it's a hamfisted ad-tech tactic deployed by ISPs that straight-up breaks a browser feature, and the browser using the DNS to unbreak it.

jeffbee 4y ago

This seems to add < 1gbps globally across all roots? Seems minor?

_qua 4y ago
Am I misreading the article or does it show that this accounts for 50% of the dns traffic the root servers see? I don’t think that’s minor.
- tptacek 4y ago
  It's the most popular implementation of what is by a very wide margin the most popular and important app on the Internet. What percentage of traffic would you have assumed it represented?
  - debugnik 4y ago
    Less than genuine lookups? Even if all the world's DNS traffic came from Chrome, probe lookups are taking 50% of all root server traffic just to confirm NXDOMAIN over and over.
    - tptacek 4y ago
      The browser confirms it at startup and when DNS settings change, not "over and over"; the "over and over" comes from the fact that zillions of people use Chrome. Serving the zillions of people who actually use the Internet is the job of the root servers. If they wanted to serve a small number of requests, they could go be the authority server for zombo.com instead.
      - deleted 4y ago
        [deleted]
  - dspillett 4y ago
    A lot less than that. The point of the hierarchy of DNS caching is to keep traffic at the root (and points of authority) low. I would assume that the vast majority of traffic to/from root DNS servers to be due to access by caching resolver daemons not individual user applications, especially not one particular user application (no matter how popular).

Aachen 4y ago

Answering the implied question in the title: ~45% of traffic to root servers is now Chromium users figuring out whether nonexistent domains get correct nxdomain responses. That's apparently 60 billion queries per day. Doing the math, a query is some 92 bytes on the wire (50 bytes UDP payload, but the root servers are probably also on ethernet or something similar so I'll include headers), so 487 megabits per second per root server is taken up by this, assuming there are no spikes and everything is perfectly averaged. Edit: and that's the downlink. I forgot the uplink traffic that will be larger due to DNS' nature of echoing the query back with the response(s) attached, plus DNSSEC... Woa. /edit. So what's the next step, Google generously stepping up to running (some of) those root servers and gaining further control and observability over what people do online? Reading further, Firefox solves this by "namespace probe queries, directing them away from the root servers towards the browser’s infrastructure".

deleted 4y ago
[deleted]
px43 4y ago
Do those numbers look big to you? Looks to me like I could host a DNS root server in my garage with a normal consumer grade gigabit fiber connection. For infrastructure hosted in a legit data center, this is nothing.
- jeffbee 4y ago
  Not only could you do so, if you did not care about reliability or latency you could host the entire DNS root on one desktop computer. Global root DNS traffic is nothing.
enneff 4y ago
Google doesn’t need to offer anything because the servers are holding up just fine under the load. These systems are trivially simple and cheap to operate by global internet standards.

lxgr 4y ago

Wow! Now I feel less bad about using "ping 8.8.8.8" as my canonical "is my internet uplink down" command…

smcl 4y ago
I think a couple of ICMP requests from a tiny subset of all internet users every now and then isn't too much strain, but if you want to lighten the load you can try alternating between Google's 8.8.8.8 and Cloudflare's 1.1.1.1 :)
trishmapow2 4y ago
CloudFlare is better since you can type ping 1.1 which becomes 1.0.0.1
- lxgr 4y ago
  Yes, but how do I know that there isn't a Cloudflare edge node somewhere in my storage cabinet, announcing anycast routes for 1.0.0.1 and 1.1.1.1 to reduce latency?

tptacek 4y ago

In almost any other scenario, this traffic would be indistinguishable from a distributed denial of service (DDoS) attack. Yes, but see, it's not a distributed denial of service attack. It's a productive use of a capability the DNS roots were designed to serve. It's a huge amount of traffic because Chrome is one of the 3 most important and popular applications on the Internet, and the most popular of those, at that. The cart doesn't drag the horse.

dpcx 4y ago
I think "productive" depends on who's paying the bill and whether the uses of it were how the system was actually designed. I don't think that the roots ever had the idea of serving random requests billions of times because that's the entire point of having the downstream servers and DNS caching. I remember a time when directly pinging the root servers was considered very bad practice because you should always be using the downstream... Google's method essentially makes those downstream servers useless.
- tptacek 4y ago
  All of these norms about DNS queries are deployed situationally. Namedroppers types angry about excess root queries today will defend the notion of end-user operating systems querying the roots directly to facilitate DNSSEC (which without direct recursive lookups from end-systems condenses down to a single "yes I did DNSSEC" bit set by your local DNS server). Everybody can look at this DNS probe code and kibitz about whether it could have been done more efficiently or whether it should be done at all, but this is a straightforward use of the DNS to do something important for end-users (detect whether their ISP's nameserver is lying to them about NXDOMAIN). Whatever else it is, it isn't a DDOS attack, and whatever else these servers are, they're not quiet out-of-the-way private systems; they're core Internet infrastructure.
  - jeffbee 4y ago
    People seem to think DNS root queries are answered by monks or something. There are ~1000 anycast sites serving the 13 roots and their load is close enough to zero to be ignored. Even if Chrome causes half of the traffic, who cares?

fwlr 4y ago

A comment on that post: “I’m the original author of this code, though I no longer maintain it. ”Just want to give folks a heads-up that we’ve been in discussion with various parties about this for some time now, as we agree the negative effects on the root servers are undesirable. So in terms of “do the Chromium folks know/care”; yes, and yes. ”This is a challenging problem to solve, as we’ve repeatedly seen in the past that NXDOMAIN hijacking is pervasive (I can’t speak quantitatively or claim that “exception rather than the norm” is false, but it’s certainly at least very common), and network operators often actively worked around previous attempts to detect it in order to ensure they could hijack without Chromium detecting it. ”We do have some proposed designs to fix; at this point much of the issue is engineering bandwidth. Though I’m sure that if ISPs wanted to cooperate with us on a reliable way to detect NXDOMAIN hijacking, we wouldn’t object. Ideally, we wouldn’t have to frustrate them, nor they us.” A way for DNS root servers and Chromium devs to cooperate that couldn’t be hijacked by domain redirecting ISPs would be nice.

sneak 4y ago
Chrome simply talking DoH to Google (or other trustworthy non-hijacking) servers by default would fix this totally.
- tptacek 4y ago
  Fix what, though? This is an insignificant load on a piece of core infrastructure for which all of the load from all of the users of the world's most popular Internet application is still a rounding error error relative to capacity. This is what those servers are there to do!
  - jeroenhd 4y ago
    Those servers are for looking up real domains, they're not designed to be a detection mechanism for malicious actors. Chrome intentionally fires unresolveable uncacheable domains at the root servers. That's dishonest use of public infrastructure at the very least. I doubt Google would be happy if Microsoft or Apple were to build a network availability tool into their systems that executes random Google queries that won't get you a result to see if searches are being redirected.

NoZebra120vClip 4y ago

Something that doesn't seem to be directly addressed here is the data-leaking nature of this thing. I don't know what a typical user of Chromium expects to happen when they type arbitrary strings into the Omnibox, but I can tell you that they probably don't expect the full text to be forwarded to several random, centralized, and highly popular DNS servers that are neither under the control of Google nor their ISPs nor their employers/institutions. This may have the effect of logging all sorts of stuff that normally Google would be hoovering up, but instead it's in the hands of third parties. Extremely trustworthy parties indeed, but they are also high-value high-stakes targets for anyone who'd want to steal valuable data in the form of query logs. Obviously there's no good solution to this. I believe Firefox led the charge with the "Awesome Bar" or whatever it was first called, which habituated me and millions of users to type search queries directly into the URL bar. It's one of those features that's so amazingly useful that it really forces the giants to carefully calculate the tradeoff costs, such as the one discussed in TFA.

m4lvin 4y ago
Wait, I thought the check here is first done with random strings, and only if those return NXDOMAIN as expected, then the real search entries that might be hostnames are sent to your configured DNS server (and only after pressing enter).
- tptacek 4y ago
  You're correct, that's what this post is about: random strings forwarded to the DNS roots to establish, without leaking any intent to the user's ISP DNS server, whether DNS NXDOMAINs are being intercepted. There's no privacy implication to this beyond the privacy benefit users get from things that push back on interception.
- deleted 4y ago
  [deleted]

Thaxll 4y ago

60B queries a day is nothing, we're talking about root servers that serve as fundation for the internet as a whole. The other thing that people would be very suprised about is how old the software is on those root server, forget modern libraries with Rust/c++ and the like, it's pretty old tech that is very inefficient. Edit: when talking about old tech I'm talking about the architecture of the DNS server used, IO libraries and models, caching, data structure and the like, for example a lot of stuff has been done arround web servers to serve things very efficiently, the same could be done on DNS servers.

nhooyr 4y ago
> it's pretty old tech that is very inefficient. There is nothing about Rust or C++ that make them faster than C. In what way are the root servers inefficient?
7steps2much 4y ago
> it's pretty old tech that is very inefficient. I can guarantee you that this "old tech" was coded with more thought invested into it than at least half of these "modern libraries". Well written C code can easily blow a C++/Rust application out of the water.
- dralley 4y ago
  Provide some, any evidence whatsoever for this assertion.
  - 7steps2much 4y ago
    > Our measurement based on the micro benchmarks shows that Rust is in general slower than C, but the extent of the slowdown varies across different programs. On average, Rust brings a 1.77x “performance overhead” compared to C. [1] Of course, for the sake of completeness it should be noted that: > With the run-time checks disabled and the restrictions loosened, Rust presents a performance indistinguishable from C. [1] Though I believe my original statement to hold none the less, as disabling these restrictions disables (amongst other things) bound and overflow checking, which is one of if not the major selling point of rust. As for C++ depends on the features that one uses. If one writes "just what one could do in C" then the machine code produced by the compiler will be exactly (almost) the same. This is due to the fact that many c++ features are only compiler relevant but compile to (almost) the same instructions as code. However, I would once again raise the question I did above with rust: If we use little to no c++ features then can we distinguish that codebasse from a c codebase in any meaningful way? But assuming we write idomatic code we will have the c++ code behaving somewhat slower due to factors such as: - automatic collections/object allocation. Datastructures growing "on demand" do in general perform slower than a comparable "none automatic" datastructure increased in larger chunks by hand (using malloc/etc. in C). While this is an implementation detail admittedly, I believe the libstdc++ does not use chunking, though I would not swear on that. - strings. While no doubt a big upgrade from \0 terminated char sequences idomatic strings in C are less efficient. Especially when it comes to concatenating or manipulation of said strings. In addition it may lead to memory fragmentation, though this should be an afterthought most of the time. In general the performance difference of C++/C comes down to "hidden" code. While by no means large, assuming software such as the dns root servers which are running essentially 24/7 and will most likely continue doing so for quite a while even small differences in performance will add up. Admittedly however my original statement of > Well written C code can easily blow a C++/Rust application out of the water. May not have been well formulated. It would have been better to split the statement and be more specific about the individual performance differences in regards to rust/c++ instead of bunching them together. [1]: https://dl.acm.org/doi/pdf/10.1145/3551349.3559494 https://dl.acm.org/doi/pdf/10.1145/3551349.3559494

jandrese 4y ago

All of this extra traffic is to support the misfeature where the URL bar and search bar are combined! They are two different functions. When I type a domain I don't want the browser to send that straight to Google and give me a search result page. It makes a little bit of sense on phones where space is tight, but on desktop browsers it's just annoying.

tptacek 4y ago
It's an extraordinarily useful feature that millions of users take advantage of every day. The idea that the operators of core Internet infrastructure would be pressuring applications not to have these kinds of features is what should alarm you, not that the features exist. The cart doesn't drag the horse.
leeoniya 4y ago
> I don't want G wants it
RockRobotRock 4y ago
You probably use Firefox anyways, so why do you care? A lot of people like this feature.
- jandrese 4y ago
  Sadly switching back to the old behavior has problems in Firefox. If you go to google.com and start typing in the search box it will redirect to the URL bar, where your search will fail because it is not a search box.

enneff 4y ago

All these indignant comments but nobody bothers to look up what happened since 2020: > In a previous blog post, we quantified that upwards of 45.80% of total DNS traffic to the root servers was, at the time, the result of Chromium intranet redirection detection tests. Since then, the Chromium team has redesigned its code to disable the redirection test on Android systems and introduced a multi-state DNS interception policy that supports disabling the redirection test for desktop browsers. This functionality was released mid-November of 2020 for Android systems in Chromium 87 and, quickly thereafter, the root servers experienced a rapid decline of DNS queries. https://blog.verisign.com/domain-names/chromiums-reduction-of-root-dns-traffic/ https://blog.verisign.com/domain-names/chromiums-reduction-o...

pkasting 4y ago
Indeed. Here's the bug where we fixed it: https://bugs.chromium.org/p/chromium/issues/detail?id=1090985 https://bugs.chromium.org/p/chromium/issues/detail?id=109098...

onphonenow 4y ago

Google would be happy to host the bandwidth for all DNS if needed. The idea that this bandwidth is relevant vs YouTube etc is crazy

tinus_hn 4y ago

Given that Chrome is the biggest browser by far, and apparently blocks these redirections, why do ISPs even bother with them?

nubinetwork 4y ago

Weird malware-like DNS queries aside, don't all browsers treat text in the URL bar that doesn't begin with http:// http:// or https:// https:// as search queries? Honestly, this is why I prefer using FQDN's, and bookmarks. I ask for what I want, and I get it (barring some captive portal, etc).