5 ms·

For me, the kicker: if I'm reading it correctly, over 40% of DNS traffic to the root server they examined is just diagnostic probes from Google Chrome being use

For me, the kicker: if I'm reading it correctly, over 40% of DNS traffic to the root server they examined is just diagnostic probes from Google Chrome being used to spot malicious DNS servers.

gbil 6y ago
We got hit by this issue in March when our remotr users increased 5+ times and the DNS traffic going through our VPNs was causing a headache to our DNS servers. We pinpointed this to tis Chrome functionality, which includes also other chromium based browers like new Edge, and we had to deploy a relevant GPO to disable this functionality. Some background, I'm talking about ~200+k remote users. Also while in the office the load is distributed in tenths of DNS servers, when on VPN only a fraction of those are used. Furthermore if I remember correctly this "feature" in chrome was enabled in a version which was distributed to our clients maybe a month before the lockdowns so there was little time to see the effect while clients were still in the office
- jiggawatts 6y ago
  The last time I saw DNS throughput or performance issues was around 2003 on a network with 200K desktops and servers. That was 17 years ago, and they don't have a problem any more, despite growing in footprint to nearly half a million client machines. I struggle to understand how DNS can possibly be a performance issue in 2020. In most corporate environments, the "working set" of a typical DNS servers will fit in the L3 cache of the CPU, or even the L2 cache. The amount of network traffic involved is similarly miniscule. If all 200K of your client machines sent 100 requests per second, each 100 bytes in size, all of those to just one server, that adds up to a paltry 2 Gbps. If your DNS servers are struggling with that, get better servers. Or networks. Or IT engineers.
  - deleted 6y ago
    [deleted]
  - deleted 6y ago
    [deleted]
- floil 6y ago
  Fairly certain this feature is more then ten years old.
  - gbil 6y ago
    https://cloud.google.com/docs/chrome-enterprise/policies/?policy=DNSInterceptionChecksEnabled https://cloud.google.com/docs/chrome-enterprise/policies/?po... Google Chrome (Linux, Mac, Windows) since version 80 Google Chrome OS (Google Chrome OS) since version 80 Chrome 80: February 4, 2020 and as a clarification When you connect via VPN to the corporate network the DNS queries are not distributes as when you are in the office. You have a X amount of entry points for the VPN which are served by Y DNS servers which is less than the total amount of DNS servers available in the corporate network. Plus the amount of remote users increased vastly, plus the VPN technology used plus the DNS servers used. Not that simple I'm afraid
    - joshuamorton 6y ago
      The article mentions relevant code changes in 2014. It seems like the enterprise policy may be recent, but the feature is much older.
      - gbil 6y ago
        Sure and I explained why it hit us when it hit us Also keep in mind that Edge is chromium based now and has the same issue. And is becoming the standard by MS and thus the impact is increased now because of this
        joshuamorton 6y ago
        Sure, lockdown and increased VPN use makes sense as to why this got painful in march. However I expect GP was quibbling with this part of your statement: >Furthermore if I remember correctly this "feature" in chrome was enabled in a version which was distributed to our clients maybe a month before the lockdowns so there was little time to see the effect while clients were still in the office Which claims that the feature was rolled out recently.
- jlgaddis 6y ago
  In a corporate / enterprise network where the DNS servers are Windows servers (domain controllers, in my experience, most of the time), the best thing you can do is stand up a few instances of <insert favorite DNS server here>, running on Linux, set them up as slaves for your internal zones, and point your users at those servers instead of your Windows servers.
  - keeperofdakeys 6y ago
    You can also use stub zones to forward traffic for a single subdomain to your AD servers, while the other dns server handles recursive queries to the internet.
- deleted 6y ago
  [deleted]
the_pwner224 6y ago
Just thinking for a few seconds, I can think of a number of ways to not repeatedly spam DNS servers while still accomplishing the objective. If you were to send 3 random queries to Google every time you opened your browser, you would quickly get hellbanned behind a recaptcha. Not saying that we should embark on some quest for retribution against Google. It's just sad.
- why_only_15 6y ago
  I bet they just made this as a throwaway and it worked fine when it was 1% of traffic and maybe they just haven't looked at it again. I bet if the APNIC people harassed them they would change it.
- thayne 6y ago
  Google could, you know, use their own DNS servers for this...
  - Slartie 6y ago
    No, they couldn't. The whole purpose of these probe requests is to assess whether the DNS server used by a particular client is acting normally (responding with NXDOMAIN if a domain does not exist), so these must bde sent to the DNS server of the client, which effectively means that unless this DNS server performs the hijacking that is to be detected, they will inevitably end up on a root DNS server, because no server in the hierarchy will know those domains. Forcing these probe requests onto Google's DNS would completely defy their purpose in the first place.
    - majewsky 6y ago
      Instead of http://asdoguhwrouyh http://asdoguhwrouyh, they could probe something like http://asdoguhwrouyh.google http://asdoguhwrouyh.google or anything else in a zone owned by them, so the uncachable traffic would hit only their authoritative name servers and not the root servers.
      - 6y ago

For me, the kicker: if I'm reading it correctly, over 40% of DNS traffic to the root server they examined is just diagnostic probes from Google Chrome being used to spot malicious DNS servers.

gbil 6y ago

We got hit by this issue in March when our remotr users increased 5+ times and the DNS traffic going through our VPNs was causing a headache to our DNS servers. We pinpointed this to tis Chrome functionality, which includes also other chromium based browers like new Edge, and we had to deploy a relevant GPO to disable this functionality. Some background, I'm talking about ~200+k remote users. Also while in the office the load is distributed in tenths of DNS servers, when on VPN only a fraction of those are used. Furthermore if I remember correctly this "feature" in chrome was enabled in a version which was distributed to our clients maybe a month before the lockdowns so there was little time to see the effect while clients were still in the office

jiggawatts 6y ago
The last time I saw DNS throughput or performance issues was around 2003 on a network with 200K desktops and servers. That was 17 years ago, and they don't have a problem any more, despite growing in footprint to nearly half a million client machines. I struggle to understand how DNS can possibly be a performance issue in 2020. In most corporate environments, the "working set" of a typical DNS servers will fit in the L3 cache of the CPU, or even the L2 cache. The amount of network traffic involved is similarly miniscule. If all 200K of your client machines sent 100 requests per second, each 100 bytes in size, all of those to just one server, that adds up to a paltry 2 Gbps. If your DNS servers are struggling with that, get better servers. Or networks. Or IT engineers.
- deleted 6y ago
  [deleted]
- deleted 6y ago
  [deleted]
floil 6y ago
Fairly certain this feature is more then ten years old.
- gbil 6y ago
  https://cloud.google.com/docs/chrome-enterprise/policies/?policy=DNSInterceptionChecksEnabled https://cloud.google.com/docs/chrome-enterprise/policies/?po... Google Chrome (Linux, Mac, Windows) since version 80 Google Chrome OS (Google Chrome OS) since version 80 Chrome 80: February 4, 2020 and as a clarification When you connect via VPN to the corporate network the DNS queries are not distributes as when you are in the office. You have a X amount of entry points for the VPN which are served by Y DNS servers which is less than the total amount of DNS servers available in the corporate network. Plus the amount of remote users increased vastly, plus the VPN technology used plus the DNS servers used. Not that simple I'm afraid
  - joshuamorton 6y ago
    The article mentions relevant code changes in 2014. It seems like the enterprise policy may be recent, but the feature is much older.
    - gbil 6y ago
      Sure and I explained why it hit us when it hit us Also keep in mind that Edge is chromium based now and has the same issue. And is becoming the standard by MS and thus the impact is increased now because of this
      - joshuamorton 6y ago
        Sure, lockdown and increased VPN use makes sense as to why this got painful in march. However I expect GP was quibbling with this part of your statement: >Furthermore if I remember correctly this "feature" in chrome was enabled in a version which was distributed to our clients maybe a month before the lockdowns so there was little time to see the effect while clients were still in the office Which claims that the feature was rolled out recently.
jlgaddis 6y ago
In a corporate / enterprise network where the DNS servers are Windows servers (domain controllers, in my experience, most of the time), the best thing you can do is stand up a few instances of <insert favorite DNS server here>, running on Linux, set them up as slaves for your internal zones, and point your users at those servers instead of your Windows servers.
- keeperofdakeys 6y ago
  You can also use stub zones to forward traffic for a single subdomain to your AD servers, while the other dns server handles recursive queries to the internet.
deleted 6y ago
[deleted]

the_pwner224 6y ago

Just thinking for a few seconds, I can think of a number of ways to not repeatedly spam DNS servers while still accomplishing the objective. If you were to send 3 random queries to Google every time you opened your browser, you would quickly get hellbanned behind a recaptcha. Not saying that we should embark on some quest for retribution against Google. It's just sad.

why_only_15 6y ago
I bet they just made this as a throwaway and it worked fine when it was 1% of traffic and maybe they just haven't looked at it again. I bet if the APNIC people harassed them they would change it.
thayne 6y ago
Google could, you know, use their own DNS servers for this...
- Slartie 6y ago
  No, they couldn't. The whole purpose of these probe requests is to assess whether the DNS server used by a particular client is acting normally (responding with NXDOMAIN if a domain does not exist), so these must bde sent to the DNS server of the client, which effectively means that unless this DNS server performs the hijacking that is to be detected, they will inevitably end up on a root DNS server, because no server in the hierarchy will know those domains. Forcing these probe requests onto Google's DNS would completely defy their purpose in the first place.
  - majewsky 6y ago
    Instead of http://asdoguhwrouyh http://asdoguhwrouyh, they could probe something like http://asdoguhwrouyh.google http://asdoguhwrouyh.google or anything else in a zone owned by them, so the uncachable traffic would hit only their authoritative name servers and not the root servers.
    - 6y ago