16 ms·
Chromium's Impact on Root DNS Traffic
- alpb 6y agoI think I've understood the most of the article but I missed the initial part. Why is there a probe in Chrome that uses DNS to query random 7-15 character long hostnames, only to get NXDOMAIN and burden the root nameservers? What does this probe achieve?
- kevingadd 6y agoSome DNS providers (like ISPs) will hijack NXDOMAINs and redirect you to ads or stuff like that. Chrome wants to detect that.
- dylz 6y agoThere was a point where, at least in the US, this was standard behaviour for virtually every single major ISP and mobile provider. Several used to hijack all port 53 traffic to disallow you from using anything but their resolver.
- ananonymoususer 6y agoAnd for those who don't understand why this is a bad thing, I will present my own use case. I run pi-hole at home and frequently work from there for another company. That company has provided me with a laptop that uses Cisco's DNS "Umbrella", which is some sort of security feature: https://docs.umbrella.com/deployment-umbrella/docs/point-your-dns-to-cisco https://docs.umbrella.com/deployment-umbrella/docs/point-you... Because my company laptop doesn't pay attention to the DNS servers recommended by DHCP, and ignores the local domain search TLD, if I try to ssh into a machine on my local network (without a FQDN) from the company laptop, it replaces the local search domain with the corporate domain, then does the lookup, and gets an A record from Umbrella that is not on my local network. It makes the ssh connection and (surprisingly) reaches an ssh server, which asks me for my password. The login fails, and my password (in plain text) could very well have been harvested by the ssh server on the catchall host. Now you are going to tell me that I shouldn't use ssh passwords, and should instead be using RSA keys for ssh. Regardless of what the NSA tells you, THIS IS ALWAYS A BAD IDEA because once any account is compromised, ALL OTHER ACCOUNTS with locally stored keys ARE ALSO COMPROMISED. Sorry for the rant, but wildcard catchall DNS is a REALLY BAD THING.
- fiskfiskfisk 6y agoIn that case you should get a host key differing message (or not present) at least.
- dylz 6y agoMy personal thought is that you shouldn't be connecting to anything personal or local from a work provided, likely heavily keylogged, device.
- noodlesUK 6y agoUser managed passwords aren’t ideal. If you’re looking for more security and you’re concerned about compromise of local keys, you could purchase a couple of yubikeys (or similar), or you could use an SSH CA (Hashicorp vault and Step come to mind). However, if you’re very concerned about storing creds on a company laptop, or compromising your passwords by logging into a honeypot server (which known_hosts should be protecting you from), you ought to be much more scared of your company keylogging you...
- zootboy 6y ago> THIS IS ALWAYS A BAD IDEA because once any account is compromised, ALL OTHER ACCOUNTS with locally stored keys ARE ALSO COMPROMISED. This is not universally true. If you generate separate private keys for each server-client pair, compromising one private key will limit the damage to just the one server.
- ananonymoususer 6y agoThat is just not true. It may be the case if the key itself is compromised, but consider that you may have many different accounts scattered on different servers. Once one of them is compromised, the attacker now has access to every other account because they are all chained together.
- shawnz 6y agoCan you describe the attack scenario you're imagining in a bit more detail? Because that doesn't sound possible to me.
- ViViDboarder 6y agoYea. They want to detect those so they can send you to their search page with their ads! How generous.
- kevingadd 6y agoFor me, the kicker: if I'm reading it correctly, over 40% of DNS traffic to the root server they examined is just diagnostic probes from Google Chrome being used to spot malicious DNS servers.
- gbil 6y agoWe got hit by this issue in March when our remotr users increased 5+ times and the DNS traffic going through our VPNs was causing a headache to our DNS servers. We pinpointed this to tis Chrome functionality, which includes also other chromium based browers like new Edge, and we had to deploy a relevant GPO to disable this functionality. Some background, I'm talking about ~200+k remote users. Also while in the office the load is distributed in tenths of DNS servers, when on VPN only a fraction of those are used. Furthermore if I remember correctly this "feature" in chrome was enabled in a version which was distributed to our clients maybe a month before the lockdowns so there was little time to see the effect while clients were still in the office
- jiggawatts 6y agoThe last time I saw DNS throughput or performance issues was around 2003 on a network with 200K desktops and servers. That was 17 years ago, and they don't have a problem any more, despite growing in footprint to nearly half a million client machines. I struggle to understand how DNS can possibly be a performance issue in 2020. In most corporate environments, the "working set" of a typical DNS servers will fit in the L3 cache of the CPU, or even the L2 cache. The amount of network traffic involved is similarly miniscule. If all 200K of your client machines sent 100 requests per second, each 100 bytes in size, all of those to just one server, that adds up to a paltry 2 Gbps. If your DNS servers are struggling with that, get better servers. Or networks. Or IT engineers.
- deleted 6y ago[deleted]
- deleted 6y ago[deleted]
- jiggawatts 6y agoI'm curious to know how much data the root namespace servers put out in terms of gbps, but this doesn't seem to be public information.
- CKN23-ARIN 6y agohttps://root-servers.org/ https://root-servers.org/ Select a root server at the bottom. Some, but not all, have a "statistics" link. Seems to be stated in qps and message size distribution, but you should be able to derive traffic volume from that.
- steventhedev 6y agoAssuming the mean is close to the median, they are reporting ~10B requests daily with a median response size of around 1KB. 10TB daily is a little under 1Gpbs. Traffic is spiky, but this isn't particularly complex once you consider they have multiple data centers/servers. Of course, I may have misread something as daily that was hourly or something like that...
- jiggawatts 6y agoSo it looks like the root namespace providers output a totally reasonable amount of traffic. Divided between the hundreds of points of presence globally, this is tens of megabits per physical host. This FAQ is illuminating: https://www.verisign.com/en_US/domain-names/internet-resolution/node-hosting/index.xhtml https://www.verisign.com/en_US/domain-names/internet-resolut... The servers themselves are ordinary 1 RU physical rack mount servers with 1 Gbps or 10 Gbps Ethernet. Nothing special. I'm guessing that most of the load isn't from the root, e.g.: "j.root-servers.net", but from hosting the authoritative DNS servers for .com and .net (b.gtld-servers.net) on the same box. That would surely have more traffic and much more data.
- moonchild 6y agoReasonable quantity of traffic, but they have to be very reliable.
- jve 6y agoThe first question I asked to myself: Is there a way to disable it? Networks i'm attached to, don't do any hijacking. And yes, luckily there is a policy to disable it: https://cloud.google.com/docs/chrome-enterprise/policies/?policy=DNSInterceptionChecksEnabled https://cloud.google.com/docs/chrome-enterprise/policies/?po... Registry key: Software\Policies\Google\Chrome\DNSInterceptionChecksEnabled PowerShell: Set-ItemProperty HKLM:\SOFTWARE\Policies\Google\Chrome -Name DNSInterceptionChecksEnabled -Value 0 -Type DWord If you are managing Chrome via GPO, you should do it via GPO. Templates can be downloaded here: https://chromeenterprise.google/browser/download/ https://chromeenterprise.google/browser/download/
- RedShift1 6y agoI wouldn't apply this policy to road warriors even if they spend most of the time in a location you have under control.
- russellbeattie 6y agoI'm sure anyone here who has set up a PiHole ad-blocking DNS server at home has run into these random domain requests and wondered what was going on. At first I thought one of my devices had a virus on it or something until I did a few searches and discovered it was Chrome being ludicrous. (Next topic: Getting Chrome to actually use the DNS provider that you specify and nothing else...)
- gzer0 6y agoWould these have occurred on a sever that has unbound as its upstream?
- jlgaddis 6y agoWhy wouldn't they? Chrome doesn't care what DNS server software is in use (even if it could figure that out), it cares whether it's behaving properly or not.
- merlinscholz 6y agoI recently just blocked port 53 in my firewall completely, for that exact reason. I use an internal DNS server the forwards to an DOH upstream server. No more rogue devices trying to use their own dns, at least until they all switch to DOH too
- samoa42 6y ago> No more rogue devices trying to use their own dns, at least until they all switch to DOH too nice that you already debunked your thesis
- nikeee 6y agoI also blocked port 53 in my firewall (except for the Pihole; no DoH there). After that, I noticed that some applications have some DNS servers hard-coded. 8.8.8.8 being pretty prominent. My solution was to assign the Pihole the IP address 8.8.8.8 as well. Then I added a static route in at the router to route 8.8.8.8 to the Pihole. Now every request to dns.google will also be handled by pihole instead of getting timeouts.
- mschuster91 6y agoThe worst thing is, this will not even detect a well written NXDOMAIN interceptor that only hijacks requests to valid top level domains. It's about time for DNSSEC to be available on all TLDs and for browsers to nag if it is broken.
- encom 6y agoThis comment, and another one mentioning DNSSEC has been downvoted. Please explain why you hate DNSSEC instead of downvoting things you disagree with.
- londons_explore 6y agoBrowser vendors seem to have shelved all work on DNSSEC for reasons they haven't publically stated. It had such promises to be able to reduce trust in CA's by pinning HTTPS certificates to DNS responses, so was exactly what browsers would have wanted, yet still all work stopped around 2015 or so. To me, it's as if DNSSEC has some critical and unfixable security vulnerability, and people who make these decisions decided to stop all work on it, but not reveal the vulnerability because doing so would do too much damage. This is probably the most comprehensive list of reasons not to use it: https://www.imperialviolet.org/2015/01/17/notdane.html https://www.imperialviolet.org/2015/01/17/notdane.html
- teddyh 6y agoThat blog post is five years old, and most of the things it lists are now moot.
- londons_explore 6y agoTrue, but dnssec is still going nowhere... The future seems to be HTTPS with domain-validated certificates over insecure DNS, or even dnssec but doing the http challenge over an insecure network. Great for state actors to inject malware into any site...
- iso1631 6y ago
- 0x0 6y agoVerisign has nobody but themselves to blame, for "inventing" this with its SiteFinder fiasco in 2003.
- djsumdog 6y agooh wow, I remember that and how it broke so many scripts and processes. It's what some of these crappy ISP DNS servers do, except for the entire .com/.net TLDs around the planet.
- aaronAgain 6y agoRight! Around 2010, when this feature was implemented in Chrome, hijacking was a business model that was discussed in regular meetings. I recall one hijacker trying to sell themselves to the company that was 'complaining' about the hijacks. "Buy us out and we'll stop, and you can use the tech on your customers?!?" One of the boldest business proposals I've been party to. After a few deep breaths and some laughter, the offer was not taken. But that wasn't a one-off event. Spent a lot of time in early 2010's directly trying to protect customers from this stuff. Still do, but it's getting much harder with TLS-everywhere, HSTS, DOH, and many other things. Not impossible though, we can never let up on the pressure to keep the ROI too low for hijacking. The various network operators and ISPs that let these companies put racks in their data-centers to inspect user traffic should be <<insert_your_own_horrible_idea_here>>.
- elric 6y agoI don't get this feature. And I really hate that it's present in pretty much every browser these days. If I want to type an URL, I'll use the address bar. If I want to search, I'll use the search bar. Different bars with different keyboard shortcuts and different purposes. Why do so many browsers merge these two? Screens are insanely wide these days, so screen real estate can't be the reason. Are we trying to trick users into thinking that URLs aren't a thing anymore? Maybe this "omnibox" doesn't know whether I want to enter a hostname or a search term, but I do.
- tasogare 6y ago> Screens are insanely wide these days This is not relevant for URL or search bars since they need to be displayed horizontally. Separate bars means less vertical screen space, which is still scarce.
- ebg13 6y ago> This is not relevant for URL or search bars since they need to be displayed horizontally Need? Has anyone tried?
- deleted 6y ago[deleted]
- tasogare 6y agoGood luck reading anything in latin script with a one-character wide vertical search bar. This would works for Chinese (that’s the traditional writing orientation) but definitely not for most other languages.
- Arnt 6y agoI've seen it, back around 1999, possibly konqueror? Something that let you drag around toolbars and if you moved the address bar to the left/right side it would change the direction of writing. Let's say that testing it briefly was enough. Editing tilted text works up to around 45 degrees, steeper than that is a strain.
- 6y ago
- 1vuio0pswjnm7 6y agoWhy does Chrome (Google) need to know whether DNS is being intercepted? What actions does Google take based on the answer? Note that under this crude test of sending queries for unregistered domains, a user who administers their own DNS could be indistingushiable from "DNS interception" by an ISP or other third party. I administer my own DNS. I do not use third party DNS. These random queries would just hit my own DNS servers, not the root servers.
- deleted 6y ago[deleted]
- jve 6y agoFrom article: > Users on such networks might be shown the “did you mean” infobar on every single-term search. To work around this, Chromium needs to know if it can trust the network to provide non-intercepted DNS responses. Don't know if this is the sole reason.
- Taek 6y agoIn my mind it's a good enough reason to justify trying to fix it.
- 1vuio0pswjnm7 6y agoI think you are right. Reminds me of the story behind "Google Public DNS". Back in 2008/2009, OpenDNS was hijacking "queries" (NXDOMAIN) typed in the address bar to their own search page ("OpenDNS Guide", or some such) on an opendns.com subdomain. In response, Google launched its own open resolver.^1 (OpenDNS was later acquired by Cisco) 1. http://umbrella.cisco.com/blog/opendns-google-dns http://umbrella.cisco.com/blog/opendns-google-dns
- ogurechny 6y agoNo, the point is that in combined address and search bar you don't know whether something is a (local) domain or a search query. You can recognize known TLDs, but that's it. Guess what Google' priorities were when they approached that problem.
- 6y ago
- peteretep 6y agoWait, so Chrome leaks the first word of my searches to my ISP? That doesn’t sound like something I want to happen
- londons_explore 6y agoChrome leaks 1 word searches in the address bar, yes.
- merlinscholz 6y agoThat's another reason to use an internal DNS server which queries an upstream DOH server.
- lstamour 6y agoLast I checked, I think “DNS Security” (DoH) was shipped in Chrome, you can pick an alternative in Settings, I think. Such as, in this case, Google. Not sure if that changes the way this nxdomain check behaves, presumably Chrome trusts TLS but not the ISP’s DoH?
- IX-103 6y agoFor DoH Chrome does not do that check. Instead one of the requirements to be one their allowed DoH providers is that they don't do the evil redirect NX Domain responses. But Chrome also falls back to try non-DoH on NX-Domain, so it doesn't really help. I guess they need to do that so internal domains work correctly.
- rsync 6y ago"That's another reason to use an internal DNS server which queries an upstream DOH server." Even better, spin up a little VM or VPS somewhere in the cloud, install 'unbound' as a recursive resolver and point it to your nextdns.io account/address. Let's unpack this ... backwards ... DNS servers out on the Internet are queried by nextdns, which presumably has no PII from you other than your CC number[1] and zip code. Nextdns receives nothing but queries from some random VPS/EC2/VM IP. Again, presumably a provider that knows (almost) nothing about you. Your ISP sees nothing ... just encrypted DNS traffic. It's win, win, win. You see no ads, since nextcloud.io acts like a pihole and strips/blocks all of the malicious hostname lookups. [1] Remember, only AMEX verifies cardholder FIRST LAST. Use your VISA/MC. I think my first/last is Nextdns User or whatever ... YMMV if a merchant is enrolled in that weird "verified by visa" service ...
- stefan_ 6y agoWhy on earth is there someone with shell access to the DNS root zone and running tcpdump?
- pilif 6y agohow would they maintain the root servers and correct issues without shell access or tcpdump? Make blind guesses and restart the server until the problem goes away (it won't)? No matter how high-profile the environment, eventually, the rubber will hit the road and some human will be in a privileged position to be able to fix a problem. That is true for every single service out there. Yes. Including Gmail. Including AWS. Including Twitter. Everywhere. Depending on size and profile of the service it's more or less people in need of jumping through more or less hoops to get there, but this must be true for any service. Always keep this in mind when you make the decision to move your data to a cloud service.
- stefan_ 6y agoWhy is a server with a problem still part of the root zone? And no, this is absolutely not the case for serious operators. Access to production systems is highly regulated.
- nickelpro 6y agoYes, highly regulated access with lots of hoop jumping, that's what they said. And there exists a person who has jumped through all the hoops and has that access. And that hoop jumping person ran tcpdump on the root server.
- gsich 6y agoHow do you remove it?
- teddyh 6y agoOK, so say you remove it, and the problem goes away. Now what do you do? How do you find out what was actually going on?
- 6y ago
- padde 6y agoIt would be interesting to have an estimate of the energy consumed (globally) by this Chrome/Chromium feature...
- jeffbee 6y agoI'm curious why you think this might be significant. All global root server traffic amounts to < 1gbps. Under contrived conditions you could easily serve it all from a single laptop computer, but even if we assume that realistically it's being served by a large, distributed collection of servers each drawing ~250 W continuously and each housed in one of those ridiculous corporate datacenters with a PUE over 2.0, you're still looking at a global energy cost comparable to one tankful of motor fuel per day, or much less than the energy used by one single commercial airplane.
- xg15 6y agoCouldn't the traffic be somewhat reduced by changing the time and order of operations? Currently, Chrome does the following: (1) on each network change, send three DNS requests with random hostnames. (1a) If at least two of the queries resolve to the same IP, store the IP as the "fake redirect address". (2) on a user search, query the first search term as DNS. (2a) If the query result is NXDOMAIN or matches the fake redirect address, do nothing. Otherwise, show the "local domain" hint. Instead, it could do: (1) on a user search, query the first search term as DNS. (1a) if the query comes back with NXDOMAIN, don't show the hint and stop. We're done. (2) otherwise, make two more DNS queries with random domain names to check for fake redirects. (2a) if the two queries resolve to the same IP as the first one, we have a fake redirect. Don't do anything. Otherwise, show the "local domain" hint. Results of step (2) could be cached until a network change. This would only require 2 instead of 3 probe queries and only if the user actually searched for something and if the search term actually caused a DNS match (fake or genuine).
- iforgotpassword 6y agoThat's pretty elegant imo. I think the NXDOMAIN of 1a can be cached too. If we get a result on the next search query it should be safe to assume it's a legit one. Maybe, at the risk of over-engineering, additionally cache the results for the last N networks persistently. Something like (gateway, DNS, localip) as key. I could see those three being identical on different networks though... And assuming the article is right and most ISPs globally do not mess with NXDOMAIN, this might not be necessary anymore with this proposal.
- skissane 6y ago> (1a) If at least two of the queries resolve to the same IP, store the IP as the "fake redirect address". From reading the source, it actually does a HTTP HEAD chasing redirects, and records the origin of the final page, and uses that as the redirect address. So even if two hostnames yield different IPs, if they end up redirecting to same hostname, it will be detected > (2a) if the two queries resolve to the same IP as the first one, we have a fake redirect. Don't do anything. Otherwise, show the "local domain" hint. What if an ISP uses multiple IPs in the fake redirect, and alternates over those IPs in each successive response?
- _qulr 6y agoOn macOS you can block these with the excellent product Little Snitch. I've got several rules for Google Chrome in Little Snitch that seem to do the trick. Deny outgoing UDP connections, and Deny outgoing TCP connections to port 80 for the IP addresses and domain for my ISP. You can see these if you monitor traffic.
- tinus_hn 6y agoFallout from the ISPs effort to hijack failed DNS queries.
- csagan5 6y agoungoogled-chromium[1] and Bromite[2] have had a patch to disable this for a while now [1] https://github.com/Eloston/ungoogled-chromium/blob/14fb2b0/patches/extra/ungoogled-chromium/disable-intranet-redirect-detector.patch https://github.com/Eloston/ungoogled-chromium/blob/14fb2b0/p... [2] https://github.com/bromite/bromite/blob/410fc50/build/patches/ungoogled-chromium-Disable-intranet-redirect-detector.patch https://github.com/bromite/bromite/blob/410fc50/build/patche...
- kevincox 6y agoIt seems like they could rotate these much less frequently to let caches work. It seems that these are random to avoid DNS servers hardcoding a response for them. However they could be pseudo random based on the current day, month or release so that it would be hard enough to intercept them (unless the DNS server was really committed to doing this, but there are other ways to achieve this) while still allowing a lot of caching. I think the only downside is that you would leak some information about your system clock.
- majewsky 6y ago> It seems that these are random to avoid DNS servers hardcoding a response for them. However they could be pseudo random based on [the current date and browser release] That would still allow ISPs to compute the limited number of domains for which NXDOMAIN would need to be sent at any given point in time. (Whether they'd do it is another story. The random pattern currently used by Chrome looks like it may still be easily detectable at the DNS-recursor level, so maybe the ISPs really don't bother beyond the simple NXDOMAIN -> portal domain replacement.)
- kevincox 6y agoAs I said, if they make specific effort they will succeed. The current scheme can be broken by returning a number of different IPs instead of one or two. I think my proposal has a nice balance between making ISPs put in non-trivial effort and not putting a lot of load on the root servers.
- aaronAgain 6y agoThis is a classic arms race. The hijackers back off for a while, but as is always the case in low-margin, low-regulation, low-consequence environments, bad actors will present a way to skim a tiny value out a massive amount of transactions. Give a percentage of that to the network operator, and take the rest home. The network operators enable this behavior. It would be next to impossible for it to be useful (ROI wise) if they didn't intentionally support it with access to their networks. It doesn't need to be an arms race, but we refuse to regulate or punish anyone in this space. We waste massive amounts of resources detecting and counteracting the hijacking services. The human (developer) cost is where the big waste is here, not electricity. and the fight goes on....
- malkia 6y agoYou can see the code online through the CS browser - https://source.chromium.org/chromium/chromium/src/+/master:chrome/browser/intranet_redirect_detector.cc;l=148?q=%22we%20generate%20a%20random%20hostname%22&ss=chromium https://source.chromium.org/chromium/chromium/src/+/master:c...
- jacobsenscott 6y agoI can't get past the `size_t i` rather than `int i` in the first loop. Why. I suppose it is some type of defensive programming.
- kevin_thibedeau 6y agoBit flip changes an int to a large negative value. Now you're stuck doing a signed comparison for a while.
- lionkor 6y agoWhy is the C++ code labelled to be coming from some file .c?