4 ms·

I see most of the bot traffic hiding behind Google and Cloudflare. I appreciate that. They take a lot of load off my servers. If I had one request of Google

I see most of the bot traffic hiding behind Google and Cloudflare. I appreciate that. They take a lot of load off my servers. If I had one request of Google it would be to remove the TTL cap of 21600 or raise it to 86400 to further reduce the traffic that comes through from them as my TTL records are very high on purpose for my own reasons that I will not debate. I know memory is still a limit. CF seem to honor my TTL's with the caveat I have to wait for each node behind their anycast clusters to cache the record but that's fine. As a side note, whatever group in Google are running DNS are doing a great job. I do not see malformed garbage coming from them. Some people watch birds when they retire, I watch packets ... and critters.

WirelessGigabit 3y ago
I aspire to be you when (if) I retire.
freedomben 3y ago
> my TTL records are very high on purpose for my own reasons that I will not debate. Why are your records high? Is the load on your servers too intense otherwise? (not trying to debate, am curious because I've really appreciated your comments/insights in the past)
- radicaldreamer 3y ago
  [flagged]
- contingencies 3y ago
  Guess - possibly some sort of outbound port-knock alternative: if (SYN->dest port && TTL > X) { ACK };
  - egberts1 3y ago
    this
  - ThePowerOfFuet 3y ago
    How is this supposed to work, exactly?
    - contingencies 3y ago
      https://en.wikipedia.org/wiki/Port_knocking https://en.wikipedia.org/wiki/Port_knocking https://wiki.archlinux.org/title/Port_knocking https://wiki.archlinux.org/title/Port_knocking
      - jsnell 3y ago
        Yes, but what does any of this have to do with TTL? Are you maybe confusing IP packet TTL with DNS record TTL?
- oooyay 3y ago
  Ask and you shall receive: https://news.ycombinator.com/item?id=21436448 https://news.ycombinator.com/item?id=21436448
- LinuxBender 3y ago
  Waited for this to fall off the front page For my personal hobby sites I change things so rarely that I can keep my TTL's very high, especially for NS records and most TXT records. There are some records I could theoretically set to 68 years though obviously nobody would keep a record that long. Going against the grain I also keep neg-ttl negative cache really high to help spot the bots that ignore neg-ttl. It's a hobby of mine to study bots and what they are enumerating. Sometimes it gives me a jump start on zero-day vulnerabilities. For example a number of bots suddenly start looking for a specific A record such as cpanel to use an old and silly example. Non bot clients will respect TTL's within reason. Most ISP recursive DNS servers will cap the TTL to 24 hours for NS records and sometimes higher for A, CNAME, PTR, etc... Most corporate DNS servers will be close to the defaults of whatever recursive daemon they are using, usually Active Directory, sometimes Bind. The remaining limiting factor is memory but most recursive DNS servers these days have obscene amounts of RAM and CPU time that the DNS admin may allocate. There are ways to even further optimize recursive servers such as periodically flushing junk zones via cron and many other things that do not need to be there as well as tuning slabs and threads based on core count. This will vary by organization and requires getting detailed zone and client statistics. An example of a junk zone would be a zone used by a corporate spy to exfiltrate data over DNS that will result in hundreds of thousands of unique A records data-flow-outbound customer or intellectual property data or TXT records inbound that are actually encrypted data data-flow-inbound encrypted malware, instrucitons. Bot scripts will bypass recursive DNS servers and will typically talk directly to authoritative servers. Most of these scripts do not even look at zone or resource record TTL times which is unfortunate for them as it makes spotting them trivial. I then dig deeper into the networks they are originating from, find their IPv4/IPv6 CIDR blocks, who they peer with, what business they claim to be. Sometimes they are squatters that are announcing routes from businesses that went under and had laid off the people that would have released the IP allocations. I help get those clawed back when I can, taking the IP allocations away from the squatters. Side project idea for Google / Cloudflare / OpenDNS / etc... These companies have a unique view into DNS traffic and could also help spot the squatters. They could free up a significant amount of IP space if they allocated a small team of interns to analyze their traffic, use automation to build graphs and reports, then once confidence is high enough submit this data to all the IP registries. Registries are also much more likely to take communication from these companies far more seriously than some retired hobbyist. This will probably come up but some may say having high TTL's is risky. It can be, but not for me. If all of the internet shared one massive /etc/hosts and DNS ceased to exist, it would rarely have updates from me and if a record is stale there would be no harm, no foul as it pertains to me. There are a myriad of other reasons I have unusually high zone/record TTL's but it would turn into a blog post and I have been too lazy as of late to make any as interest is usually very low. I don't even have my blog VM's spun up.
yuliyp 3y ago
A recursive resolver is not a database. They're under no obligation to cache for a day.
- Waterluvian 3y ago
  Okay stupid question… I thought TTL was a count of hops, not time in seconds. Or do people use it differently where instead of subtracting 1, their routing devices subtract n seconds?
  - yuliyp 3y ago
    TTL is a general term standing for "time to live". In the context of an IP packet it's a number of hops. In the context of a cache, it's a time until expiration.
    - Waterluvian 3y ago
      Derp. Right. We’re talking about caching.
    - ongy 3y ago
      IP is weird. TTL is still in seconds, but every hop has to decrease it by at least one. I'm not sure if there is any implementation out there that cares, but if e.g. wifi retries or a massive buffer queue lead to a packet spending more than a second on a hop, it should decrease by two.
      - dspillett 3y ago
        TTL for a DNS response is in seconds, and is intended to be the maximum time a value is cached. If you request an address for a name that the server you are talking to is authoritative for, then you will always get the TTL set in the DNS records. If your request goes to a caching name server that is not authoritative then you may get a lower value, in fact you most likely will. If the intermediate server makes a new request upstream it can, as per the spec, just pass on the TTL it gets¹, but some seem to drop one second just-in-case even though it isn't needed² really. If an intermediate server handed out the original TTL for all requests then the TTL would be effectively multiplied by each caching layer³, which is why you will only get the true TTL at the client end if a fresh request was actually made to the authoritative server(s). -- [1] the fraction of a second difference caused by latency isn't likely to be significant for either long or short TTLs [2] and on a really high latency link (like back when GPRS or POTS landlines were common, or some links even now in deep rural areas) 1s might not be enough anyway for the benefit they think they are giving [3] so at my home where lookups go PiHole->8.8.8.8->authoritative it would make TTLs potentially up to 2x their intended length. Example: for a 1000s TTL if PiHole gets the lookup request with 1s remaining but hands out the true TTL, my client will not check for 1000s, potentially 999s late. Depending on the sequence of related requests to each DNS server from other clients, that could become 1998s in my case, longer if there were more caching layers.

WirelessGigabit 3y ago

I aspire to be you when (if) I retire.

freedomben 3y ago

> my TTL records are very high on purpose for my own reasons that I will not debate. Why are your records high? Is the load on your servers too intense otherwise? (not trying to debate, am curious because I've really appreciated your comments/insights in the past)

radicaldreamer 3y ago
[flagged]
contingencies 3y ago
Guess - possibly some sort of outbound port-knock alternative: if (SYN->dest port && TTL > X) { ACK };
- egberts1 3y ago
  this
- ThePowerOfFuet 3y ago
  How is this supposed to work, exactly?
  - contingencies 3y ago
    https://en.wikipedia.org/wiki/Port_knocking https://en.wikipedia.org/wiki/Port_knocking https://wiki.archlinux.org/title/Port_knocking https://wiki.archlinux.org/title/Port_knocking
    - jsnell 3y ago
      Yes, but what does any of this have to do with TTL? Are you maybe confusing IP packet TTL with DNS record TTL?
oooyay 3y ago
Ask and you shall receive: https://news.ycombinator.com/item?id=21436448 https://news.ycombinator.com/item?id=21436448
LinuxBender 3y ago
Waited for this to fall off the front page For my personal hobby sites I change things so rarely that I can keep my TTL's very high, especially for NS records and most TXT records. There are some records I could theoretically set to 68 years though obviously nobody would keep a record that long. Going against the grain I also keep neg-ttl negative cache really high to help spot the bots that ignore neg-ttl. It's a hobby of mine to study bots and what they are enumerating. Sometimes it gives me a jump start on zero-day vulnerabilities. For example a number of bots suddenly start looking for a specific A record such as cpanel to use an old and silly example. Non bot clients will respect TTL's within reason. Most ISP recursive DNS servers will cap the TTL to 24 hours for NS records and sometimes higher for A, CNAME, PTR, etc... Most corporate DNS servers will be close to the defaults of whatever recursive daemon they are using, usually Active Directory, sometimes Bind. The remaining limiting factor is memory but most recursive DNS servers these days have obscene amounts of RAM and CPU time that the DNS admin may allocate. There are ways to even further optimize recursive servers such as periodically flushing junk zones via cron and many other things that do not need to be there as well as tuning slabs and threads based on core count. This will vary by organization and requires getting detailed zone and client statistics. An example of a junk zone would be a zone used by a corporate spy to exfiltrate data over DNS that will result in hundreds of thousands of unique A records data-flow-outbound customer or intellectual property data or TXT records inbound that are actually encrypted data data-flow-inbound encrypted malware, instrucitons. Bot scripts will bypass recursive DNS servers and will typically talk directly to authoritative servers. Most of these scripts do not even look at zone or resource record TTL times which is unfortunate for them as it makes spotting them trivial. I then dig deeper into the networks they are originating from, find their IPv4/IPv6 CIDR blocks, who they peer with, what business they claim to be. Sometimes they are squatters that are announcing routes from businesses that went under and had laid off the people that would have released the IP allocations. I help get those clawed back when I can, taking the IP allocations away from the squatters. Side project idea for Google / Cloudflare / OpenDNS / etc... These companies have a unique view into DNS traffic and could also help spot the squatters. They could free up a significant amount of IP space if they allocated a small team of interns to analyze their traffic, use automation to build graphs and reports, then once confidence is high enough submit this data to all the IP registries. Registries are also much more likely to take communication from these companies far more seriously than some retired hobbyist. This will probably come up but some may say having high TTL's is risky. It can be, but not for me. If all of the internet shared one massive /etc/hosts and DNS ceased to exist, it would rarely have updates from me and if a record is stale there would be no harm, no foul as it pertains to me. There are a myriad of other reasons I have unusually high zone/record TTL's but it would turn into a blog post and I have been too lazy as of late to make any as interest is usually very low. I don't even have my blog VM's spun up.

yuliyp 3y ago

A recursive resolver is not a database. They're under no obligation to cache for a day.

Waterluvian 3y ago
Okay stupid question… I thought TTL was a count of hops, not time in seconds. Or do people use it differently where instead of subtracting 1, their routing devices subtract n seconds?
- yuliyp 3y ago
  TTL is a general term standing for "time to live". In the context of an IP packet it's a number of hops. In the context of a cache, it's a time until expiration.
  - Waterluvian 3y ago
    Derp. Right. We’re talking about caching.
  - ongy 3y ago
    IP is weird. TTL is still in seconds, but every hop has to decrease it by at least one. I'm not sure if there is any implementation out there that cares, but if e.g. wifi retries or a massive buffer queue lead to a packet spending more than a second on a hop, it should decrease by two.
    - dspillett 3y ago
      TTL for a DNS response is in seconds, and is intended to be the maximum time a value is cached. If you request an address for a name that the server you are talking to is authoritative for, then you will always get the TTL set in the DNS records. If your request goes to a caching name server that is not authoritative then you may get a lower value, in fact you most likely will. If the intermediate server makes a new request upstream it can, as per the spec, just pass on the TTL it gets¹, but some seem to drop one second just-in-case even though it isn't needed² really. If an intermediate server handed out the original TTL for all requests then the TTL would be effectively multiplied by each caching layer³, which is why you will only get the true TTL at the client end if a fresh request was actually made to the authoritative server(s). -- [1] the fraction of a second difference caused by latency isn't likely to be significant for either long or short TTLs [2] and on a really high latency link (like back when GPRS or POTS landlines were common, or some links even now in deep rural areas) 1s might not be enough anyway for the benefit they think they are giving [3] so at my home where lookups go PiHole->8.8.8.8->authoritative it would make TTLs potentially up to 2x their intended length. Example: for a 1000s TTL if PiHole gets the lookup request with 1s remaining but hands out the true TTL, my client will not check for 1000s, potentially 999s late. Depending on the sequence of related requests to each DNS server from other clients, that could become 1998s in my case, longer if there were more caching layers.