8 ms·

Google Public DNS's approach to fight against cache poisoning attacks

212 points by tatersolid 3y ago

LinuxBender 3y ago
I see most of the bot traffic hiding behind Google and Cloudflare. I appreciate that. They take a lot of load off my servers. If I had one request of Google it would be to remove the TTL cap of 21600 or raise it to 86400 to further reduce the traffic that comes through from them as my TTL records are very high on purpose for my own reasons that I will not debate. I know memory is still a limit. CF seem to honor my TTL's with the caveat I have to wait for each node behind their anycast clusters to cache the record but that's fine. As a side note, whatever group in Google are running DNS are doing a great job. I do not see malformed garbage coming from them. Some people watch birds when they retire, I watch packets ... and critters.
- WirelessGigabit 3y ago
  I aspire to be you when (if) I retire.
- freedomben 3y ago
  > my TTL records are very high on purpose for my own reasons that I will not debate. Why are your records high? Is the load on your servers too intense otherwise? (not trying to debate, am curious because I've really appreciated your comments/insights in the past)
  - radicaldreamer 3y ago
    [flagged]
  - contingencies 3y ago
    Guess - possibly some sort of outbound port-knock alternative: if (SYN->dest port && TTL > X) { ACK };
    - egberts1 3y ago
      this
    - ThePowerOfFuet 3y ago
      How is this supposed to work, exactly?
      - contingencies 3y ago
        https://en.wikipedia.org/wiki/Port_knocking https://en.wikipedia.org/wiki/Port_knocking https://wiki.archlinux.org/title/Port_knocking https://wiki.archlinux.org/title/Port_knocking
        jsnell 3y ago
        Yes, but what does any of this have to do with TTL? Are you maybe confusing IP packet TTL with DNS record TTL?
  - oooyay 3y ago
    Ask and you shall receive: https://news.ycombinator.com/item?id=21436448 https://news.ycombinator.com/item?id=21436448
  - LinuxBender 3y ago
    Waited for this to fall off the front page For my personal hobby sites I change things so rarely that I can keep my TTL's very high, especially for NS records and most TXT records. There are some records I could theoretically set to 68 years though obviously nobody would keep a record that long. Going against the grain I also keep neg-ttl negative cache really high to help spot the bots that ignore neg-ttl. It's a hobby of mine to study bots and what they are enumerating. Sometimes it gives me a jump start on zero-day vulnerabilities. For example a number of bots suddenly start looking for a specific A record such as cpanel to use an old and silly example. Non bot clients will respect TTL's within reason. Most ISP recursive DNS servers will cap the TTL to 24 hours for NS records and sometimes higher for A, CNAME, PTR, etc... Most corporate DNS servers will be close to the defaults of whatever recursive daemon they are using, usually Active Directory, sometimes Bind. The remaining limiting factor is memory but most recursive DNS servers these days have obscene amounts of RAM and CPU time that the DNS admin may allocate. There are ways to even further optimize recursive servers such as periodically flushing junk zones via cron and many other things that do not need to be there as well as tuning slabs and threads based on core count. This will vary by organization and requires getting detailed zone and client statistics. An example of a junk zone would be a zone used by a corporate spy to exfiltrate data over DNS that will result in hundreds of thousands of unique A records data-flow-outbound customer or intellectual property data or TXT records inbound that are actually encrypted data data-flow-inbound encrypted malware, instrucitons. Bot scripts will bypass recursive DNS servers and will typically talk directly to authoritative servers. Most of these scripts do not even look at zone or resource record TTL times which is unfortunate for them as it makes spotting them trivial. I then dig deeper into the networks they are originating from, find their IPv4/IPv6 CIDR blocks, who they peer with, what business they claim to be. Sometimes they are squatters that are announcing routes from businesses that went under and had laid off the people that would have released the IP allocations. I help get those clawed back when I can, taking the IP allocations away from the squatters. Side project idea for Google / Cloudflare / OpenDNS / etc... These companies have a unique view into DNS traffic and could also help spot the squatters. They could free up a significant amount of IP space if they allocated a small team of interns to analyze their traffic, use automation to build graphs and reports, then once confidence is high enough submit this data to all the IP registries. Registries are also much more likely to take communication from these companies far more seriously than some retired hobbyist. This will probably come up but some may say having high TTL's is risky. It can be, but not for me. If all of the internet shared one massive /etc/hosts and DNS ceased to exist, it would rarely have updates from me and if a record is stale there would be no harm, no foul as it pertains to me. There are a myriad of other reasons I have unusually high zone/record TTL's but it would turn into a blog post and I have been too lazy as of late to make any as interest is usually very low. I don't even have my blog VM's spun up.
- yuliyp 3y ago
  A recursive resolver is not a database. They're under no obligation to cache for a day.
  - Waterluvian 3y ago
    Okay stupid question… I thought TTL was a count of hops, not time in seconds. Or do people use it differently where instead of subtracting 1, their routing devices subtract n seconds?
    - yuliyp 3y ago
      TTL is a general term standing for "time to live". In the context of an IP packet it's a number of hops. In the context of a cache, it's a time until expiration.
      - Waterluvian 3y ago
        Derp. Right. We’re talking about caching.
      - ongy 3y ago
        IP is weird. TTL is still in seconds, but every hop has to decrease it by at least one. I'm not sure if there is any implementation out there that cares, but if e.g. wifi retries or a massive buffer queue lead to a packet spending more than a second on a hop, it should decrease by two.
        dspillett 3y ago
        TTL for a DNS response is in seconds, and is intended to be the maximum time a value is cached. If you request an address for a name that the server you are talking to is authoritative for, then you will always get the TTL set in the DNS records. If your request goes to a caching name server that is not authoritative then you may get a lower value, in fact you most likely will. If the intermediate server makes a new request upstream it can, as per the spec, just pass on the TTL it gets¹, but some seem to drop one second just-in-case even though it isn't needed² really. If an intermediate server handed out the original TTL for all requests then the TTL would be effectively multiplied by each caching layer³, which is why you will only get the true TTL at the client end if a fresh request was actually made to the authoritative server(s). -- [1] the fraction of a second difference caused by latency isn't likely to be significant for either long or short TTLs [2] and on a really high latency link (like back when GPRS or POTS landlines were common, or some links even now in deep rural areas) 1s might not be enough anyway for the benefit they think they are giving [3] so at my home where lookups go PiHole->8.8.8.8->authoritative it would make TTLs potentially up to 2x their intended length. Example: for a 1000s TTL if PiHole gets the lookup request with 1s remaining but hands out the true TTL, my client will not check for 1000s, potentially 999s late. Depending on the sequence of related requests to each DNS server from other clients, that could become 1998s in my case, longer if there were more caching layers.
ape4 3y ago
Good to hear the world's infrastructure relies on a kludge like case randomization.
- bradfitz 3y ago
  Longer domain names are more secure!
  - jamespwilliams 3y ago
    And conversely, short domains like Google’s own g.co are less secure!
- akira2501 3y ago
  That's how you maintain backwards compatibility while improving security. I'm not sure lamenting the imperfection is valuable, but it is a worthwhile lesson for those designing new protocols. If it becomes popular enough you will certainly face future security challenges you failed to even imagine. Leave some room for that. Otherwise, this is great work.
AndyMcConachie 3y ago
Weird that they don't even mention that Google's public DNS performs DNSSEC validation. It does, and that's the ultimate defense against cache poisoning attacks.
- tptacek 3y ago
  No, it obviously isn't, because less than 5% of zones are signed, and an even smaller fraction of important zones (the Moz 500, the Tranco list) are. That's why they don't mention DNSSEC: because it isn't a significant security mechanism for Internet DNS. It's also why they do mention ADoT: because it is. I think this is also why DNSSEC advocates are so fixated on DANE, which is (necessarily) an even harder lift than getting DNSSEC deployed: because the attacks DNSSEC were ostensibly designed to address are now solved problems. Note also that if ADoT rolls out all the way --- it's already significantly more available than DNSSEC! --- there won't even be a real architectural argument for DNSSEC anymore, because we'll have end-to-end cryptographic security for the DNS. Thanks for calling this out! That Google feels case randomization is more important than DNSSEC is indeed telling.
  - tialaramex 3y ago
    > the attacks DNSSEC were ostensibly designed to address are now solved problems. Maybe you can help figure out where you went wrong here by explaining what - in your understanding - were the problems that DNSSEC was "ostensibly designed to address" ? > because we'll have end-to-end cryptographic security for the DNS. In 1995 a researcher who was annoyed about people snooping his passwords over telnet invented a protocol (and gave away a free Unix program) which I guess you'd say delivers "end-to-end cryptographic security" for the remote shell. Now, when you go into a startup and you find they've set up a bunch of ad hoc SSH servers and their people are just agreeing to all the "Are you sure you want to continue ..?" messages, do you think "That's fine, it's end-to-end cryptographic security" ? Or do you immediately put that on the Must Do list for basic security because it's an obvious vulnerability ?
  - Hazelnut2465 3y ago
    > there won't even be a real architectural argument for DNSSEC anymore ADoT relies on NS records to be DNSSEC signed. The TLS certificates that ADoT relies on need to be hashed into TLSA records (DANE, DNSSEC).
xchip 3y ago
[flagged]
mjl- 3y ago
Wondering what prompted the blog post. The recent publication of RFC 9539? It would be interesting to hear how often the google dns servers see attempts to poison their cache. The mitigations probably prevent folks from even trying, but he numbers would be interesting. The OARC 40 presentation PDF mentions cookies deployment is low for large operators but open source software has compliant implementations. Are large operators writing their own dns servers, but badly? I would think there wouldn't be many custom implementations, and that you would be able to detect which software nameservers are running, each with known capabilities. But from the way the numbers are presented it seems they only look at behaviour without considering software (versions).
- mike_d 3y ago
  > Are large operators writing their own dns servers, but badly? Yes. It is trivial to build a DNS server, and near impossible to write a correct DNS server. Eventually your organization gets large enough that someone thinks it is a good idea without understanding the implications. > you would be able to detect which software nameservers are running, each with known capabilities There are pseudo-standards for asking an authoritative server what software it is running, but everyone turns that off because somehow it makes you "more secure." What you end up having to do is probe auth servers by replaying user queries on the side, measuring if the responses you get are correct, and then keeping a database somewhere of which servers support which flags.
rainsford 3y ago
I'm a little surprised Google only implemented case randomization by default in 2022 considering it's been around since 2008. Presumably they had concerns about widespread compatibility? Although my understanding is that for a lot of DNS servers it just worked without any specific implementation effort...but maybe there was a long tail of random server types Google was concerned about.
spydum 3y ago
This is so weird to see. Just this morning I was checking thru my public authoritative NS query logs, and noticed the random capitalization. I had also noticed this in a similar work environment roughly end of 2023, but attributed it to people just doing DNS wordlist bruteforcing to find stuff (couldn't explain the case, but figured it was some evasion). Today I let my curiosity dive deeper and quickly found the ietf publication on 0x20 encoding and this article. Just odd to see others post it to hn on the same day.. coincidences are weird.
jeffbee 3y ago
I first heard about this 0x20 scheme around 2015 when I was working on a DNS cache (also at Google, but not for the public DNS team). I noticed and had to work around the fact that some servers were responding in vixie-case even when the requests were not. Those servers would be broken if the requests were paying attention to 0x20, right? I wonder what software was doing that.
MarkSweep 3y ago
The that the longer your domain name, the less susceptible is it to cache poisoning attacks, right? Since there are more possible case variations.
mleonhard 3y ago
I updated https://crates.io/crates/dns-server https://crates.io/crates/dns-server to support case randomization.

212 points by tatersolid 3y ago

I see most of the bot traffic hiding behind Google and Cloudflare. I appreciate that. They take a lot of load off my servers. If I had one request of Google it would be to remove the TTL cap of 21600 or raise it to 86400 to further reduce the traffic that comes through from them as my TTL records are very high on purpose for my own reasons that I will not debate. I know memory is still a limit. CF seem to honor my TTL's with the caveat I have to wait for each node behind their anycast clusters to cache the record but that's fine. As a side note, whatever group in Google are running DNS are doing a great job. I do not see malformed garbage coming from them. Some people watch birds when they retire, I watch packets ... and critters.

WirelessGigabit 3y ago
I aspire to be you when (if) I retire.
freedomben 3y ago
> my TTL records are very high on purpose for my own reasons that I will not debate. Why are your records high? Is the load on your servers too intense otherwise? (not trying to debate, am curious because I've really appreciated your comments/insights in the past)
- radicaldreamer 3y ago
  [flagged]
- contingencies 3y ago
  Guess - possibly some sort of outbound port-knock alternative: if (SYN->dest port && TTL > X) { ACK };
  - egberts1 3y ago
    this
  - ThePowerOfFuet 3y ago
    How is this supposed to work, exactly?
    - contingencies 3y ago
      https://en.wikipedia.org/wiki/Port_knocking https://en.wikipedia.org/wiki/Port_knocking https://wiki.archlinux.org/title/Port_knocking https://wiki.archlinux.org/title/Port_knocking
      - jsnell 3y ago
        Yes, but what does any of this have to do with TTL? Are you maybe confusing IP packet TTL with DNS record TTL?
- oooyay 3y ago
  Ask and you shall receive: https://news.ycombinator.com/item?id=21436448 https://news.ycombinator.com/item?id=21436448
- LinuxBender 3y ago
  Waited for this to fall off the front page For my personal hobby sites I change things so rarely that I can keep my TTL's very high, especially for NS records and most TXT records. There are some records I could theoretically set to 68 years though obviously nobody would keep a record that long. Going against the grain I also keep neg-ttl negative cache really high to help spot the bots that ignore neg-ttl. It's a hobby of mine to study bots and what they are enumerating. Sometimes it gives me a jump start on zero-day vulnerabilities. For example a number of bots suddenly start looking for a specific A record such as cpanel to use an old and silly example. Non bot clients will respect TTL's within reason. Most ISP recursive DNS servers will cap the TTL to 24 hours for NS records and sometimes higher for A, CNAME, PTR, etc... Most corporate DNS servers will be close to the defaults of whatever recursive daemon they are using, usually Active Directory, sometimes Bind. The remaining limiting factor is memory but most recursive DNS servers these days have obscene amounts of RAM and CPU time that the DNS admin may allocate. There are ways to even further optimize recursive servers such as periodically flushing junk zones via cron and many other things that do not need to be there as well as tuning slabs and threads based on core count. This will vary by organization and requires getting detailed zone and client statistics. An example of a junk zone would be a zone used by a corporate spy to exfiltrate data over DNS that will result in hundreds of thousands of unique A records data-flow-outbound customer or intellectual property data or TXT records inbound that are actually encrypted data data-flow-inbound encrypted malware, instrucitons. Bot scripts will bypass recursive DNS servers and will typically talk directly to authoritative servers. Most of these scripts do not even look at zone or resource record TTL times which is unfortunate for them as it makes spotting them trivial. I then dig deeper into the networks they are originating from, find their IPv4/IPv6 CIDR blocks, who they peer with, what business they claim to be. Sometimes they are squatters that are announcing routes from businesses that went under and had laid off the people that would have released the IP allocations. I help get those clawed back when I can, taking the IP allocations away from the squatters. Side project idea for Google / Cloudflare / OpenDNS / etc... These companies have a unique view into DNS traffic and could also help spot the squatters. They could free up a significant amount of IP space if they allocated a small team of interns to analyze their traffic, use automation to build graphs and reports, then once confidence is high enough submit this data to all the IP registries. Registries are also much more likely to take communication from these companies far more seriously than some retired hobbyist. This will probably come up but some may say having high TTL's is risky. It can be, but not for me. If all of the internet shared one massive /etc/hosts and DNS ceased to exist, it would rarely have updates from me and if a record is stale there would be no harm, no foul as it pertains to me. There are a myriad of other reasons I have unusually high zone/record TTL's but it would turn into a blog post and I have been too lazy as of late to make any as interest is usually very low. I don't even have my blog VM's spun up.
yuliyp 3y ago
A recursive resolver is not a database. They're under no obligation to cache for a day.
- Waterluvian 3y ago
  Okay stupid question… I thought TTL was a count of hops, not time in seconds. Or do people use it differently where instead of subtracting 1, their routing devices subtract n seconds?
  - yuliyp 3y ago
    TTL is a general term standing for "time to live". In the context of an IP packet it's a number of hops. In the context of a cache, it's a time until expiration.
    - Waterluvian 3y ago
      Derp. Right. We’re talking about caching.
    - ongy 3y ago
      IP is weird. TTL is still in seconds, but every hop has to decrease it by at least one. I'm not sure if there is any implementation out there that cares, but if e.g. wifi retries or a massive buffer queue lead to a packet spending more than a second on a hop, it should decrease by two.
      - dspillett 3y ago
        TTL for a DNS response is in seconds, and is intended to be the maximum time a value is cached. If you request an address for a name that the server you are talking to is authoritative for, then you will always get the TTL set in the DNS records. If your request goes to a caching name server that is not authoritative then you may get a lower value, in fact you most likely will. If the intermediate server makes a new request upstream it can, as per the spec, just pass on the TTL it gets¹, but some seem to drop one second just-in-case even though it isn't needed² really. If an intermediate server handed out the original TTL for all requests then the TTL would be effectively multiplied by each caching layer³, which is why you will only get the true TTL at the client end if a fresh request was actually made to the authoritative server(s). -- [1] the fraction of a second difference caused by latency isn't likely to be significant for either long or short TTLs [2] and on a really high latency link (like back when GPRS or POTS landlines were common, or some links even now in deep rural areas) 1s might not be enough anyway for the benefit they think they are giving [3] so at my home where lookups go PiHole->8.8.8.8->authoritative it would make TTLs potentially up to 2x their intended length. Example: for a 1000s TTL if PiHole gets the lookup request with 1s remaining but hands out the true TTL, my client will not check for 1000s, potentially 999s late. Depending on the sequence of related requests to each DNS server from other clients, that could become 1998s in my case, longer if there were more caching layers.

ape4 3y ago

Good to hear the world's infrastructure relies on a kludge like case randomization.

bradfitz 3y ago
Longer domain names are more secure!
- jamespwilliams 3y ago
  And conversely, short domains like Google’s own g.co are less secure!
akira2501 3y ago
That's how you maintain backwards compatibility while improving security. I'm not sure lamenting the imperfection is valuable, but it is a worthwhile lesson for those designing new protocols. If it becomes popular enough you will certainly face future security challenges you failed to even imagine. Leave some room for that. Otherwise, this is great work.

AndyMcConachie 3y ago

Weird that they don't even mention that Google's public DNS performs DNSSEC validation. It does, and that's the ultimate defense against cache poisoning attacks.

tptacek 3y ago
No, it obviously isn't, because less than 5% of zones are signed, and an even smaller fraction of important zones (the Moz 500, the Tranco list) are. That's why they don't mention DNSSEC: because it isn't a significant security mechanism for Internet DNS. It's also why they do mention ADoT: because it is. I think this is also why DNSSEC advocates are so fixated on DANE, which is (necessarily) an even harder lift than getting DNSSEC deployed: because the attacks DNSSEC were ostensibly designed to address are now solved problems. Note also that if ADoT rolls out all the way --- it's already significantly more available than DNSSEC! --- there won't even be a real architectural argument for DNSSEC anymore, because we'll have end-to-end cryptographic security for the DNS. Thanks for calling this out! That Google feels case randomization is more important than DNSSEC is indeed telling.
- tialaramex 3y ago
  > the attacks DNSSEC were ostensibly designed to address are now solved problems. Maybe you can help figure out where you went wrong here by explaining what - in your understanding - were the problems that DNSSEC was "ostensibly designed to address" ? > because we'll have end-to-end cryptographic security for the DNS. In 1995 a researcher who was annoyed about people snooping his passwords over telnet invented a protocol (and gave away a free Unix program) which I guess you'd say delivers "end-to-end cryptographic security" for the remote shell. Now, when you go into a startup and you find they've set up a bunch of ad hoc SSH servers and their people are just agreeing to all the "Are you sure you want to continue ..?" messages, do you think "That's fine, it's end-to-end cryptographic security" ? Or do you immediately put that on the Must Do list for basic security because it's an obvious vulnerability ?
- Hazelnut2465 3y ago
  > there won't even be a real architectural argument for DNSSEC anymore ADoT relies on NS records to be DNSSEC signed. The TLS certificates that ADoT relies on need to be hashed into TLSA records (DANE, DNSSEC).

xchip 3y ago

[flagged]

mjl- 3y ago

Wondering what prompted the blog post. The recent publication of RFC 9539? It would be interesting to hear how often the google dns servers see attempts to poison their cache. The mitigations probably prevent folks from even trying, but he numbers would be interesting. The OARC 40 presentation PDF mentions cookies deployment is low for large operators but open source software has compliant implementations. Are large operators writing their own dns servers, but badly? I would think there wouldn't be many custom implementations, and that you would be able to detect which software nameservers are running, each with known capabilities. But from the way the numbers are presented it seems they only look at behaviour without considering software (versions).

mike_d 3y ago
> Are large operators writing their own dns servers, but badly? Yes. It is trivial to build a DNS server, and near impossible to write a correct DNS server. Eventually your organization gets large enough that someone thinks it is a good idea without understanding the implications. > you would be able to detect which software nameservers are running, each with known capabilities There are pseudo-standards for asking an authoritative server what software it is running, but everyone turns that off because somehow it makes you "more secure." What you end up having to do is probe auth servers by replaying user queries on the side, measuring if the responses you get are correct, and then keeping a database somewhere of which servers support which flags.

rainsford 3y ago

I'm a little surprised Google only implemented case randomization by default in 2022 considering it's been around since 2008. Presumably they had concerns about widespread compatibility? Although my understanding is that for a lot of DNS servers it just worked without any specific implementation effort...but maybe there was a long tail of random server types Google was concerned about.

spydum 3y ago

This is so weird to see. Just this morning I was checking thru my public authoritative NS query logs, and noticed the random capitalization. I had also noticed this in a similar work environment roughly end of 2023, but attributed it to people just doing DNS wordlist bruteforcing to find stuff (couldn't explain the case, but figured it was some evasion). Today I let my curiosity dive deeper and quickly found the ietf publication on 0x20 encoding and this article. Just odd to see others post it to hn on the same day.. coincidences are weird.

jeffbee 3y ago

I first heard about this 0x20 scheme around 2015 when I was working on a DNS cache (also at Google, but not for the public DNS team). I noticed and had to work around the fact that some servers were responding in vixie-case even when the requests were not. Those servers would be broken if the requests were paying attention to 0x20, right? I wonder what software was doing that.

MarkSweep 3y ago

The that the longer your domain name, the less susceptible is it to cache poisoning attacks, right? Since there are more possible case variations.

mleonhard 3y ago

I updated https://crates.io/crates/dns-server https://crates.io/crates/dns-server to support case randomization.