12 ms·
What happens when you update your DNS
- eat_veggies 6y agoOne of my favorite revelations about the network tracing tools (things like `traceroute` and `dig +trace`) that might not be obvious for people like me who work higher up in the stack, is that the data they provide isn't usually made available during "normal" usage. Packets don't just phone home and tell you where they've been. Something else is going on. When you send a DNS query to a recursive server like your ISP's or something like 1.1.1.1, you make a single DNS query and get back a single response, because the recursive DNS server handles all the different queries that Julia outlines in the post. As the client, we have no idea what steps just happened in the background. But when you run `dig +trace`, dig is actually pretending to be a recursive name server, and making all those queries itself instead of letting the real recursive name servers do their work. It's a fun hack but that means it's not always 100% accurate to what's going on in the real world [0] [0] https://serverfault.com/questions/482913/is-dig-trace-always-accurate https://serverfault.com/questions/482913/is-dig-trace-always...
- nijave 6y agoYup, and to complicate matters more those resolvers you're talking to may be talking to more caching resolvers. For a given application server it might be: - check local dns caching resolver - check local network caching resolver - if internal domain, check local authoritative resolver - if public domain check isp resolver - recursively resolve from there
- fragmede 6y agoIn particular, one your ISPs/their ISP's DNS servers may be caching a record for longer than it's supposed to and will return incorrect and expired data. The other possibility is different IP's being returned by a DNS server based on where a query is coming from, eg a CDN. If you're in location A and your ISPs DNS server is in location B, the CDN's DNS server may return a different IP based on if the request is coming from A or B. ECS [0] is supposed to mitigate this, but may or may not be used. [0] https://en.wikipedia.org/wiki/EDNS_Client_Subnet https://en.wikipedia.org/wiki/EDNS_Client_Subnet
- qes 6y ago> In particular, one your ISPs/their ISP's DNS servers may be caching a record for longer than it's supposed to and will return incorrect and expired data. It's disturbing how many clients we'll see hitting an old IP address for 30 days after a change.
- dgl 6y agodig +trace takes one path, there’s also tools like dnstrace that attempt to show all the paths: https://github.com/rs/dnstrace https://github.com/rs/dnstrace Still there can be caches that don’t quite agree as the other comment mentions.
- LogicX 6y agoJust to add to the discussion -- 'whats happening in the background' -- more specifically is your operating system's stub resolver. So when you ask for www.amazon.com it ends up making multiple DNS lookups, as www.amazon.com is a CNAME record. Nothing about this CNAME lookup gets passed back up the stack to your application; you just get that end-result: the IP address. host www.amazon.com www.amazon.com is an alias for tp.47cf2c8c9-frontier.amazon.com. tp.47cf2c8c9-frontier.amazon.com is an alias for www.amazon.com.edgekey.net. www.amazon.com.edgekey.net is an alias for e15316.e22.akamaiedge.net. e15316.e22.akamaiedge.net has address 23.204.68.114
- chrisweekly 6y agoAm I the only one who finds it strange that Amazon uses Akamai instead of Cloudfront?
- _jal 6y agoWhy is that? (My first thought was, wow, surprised they outsource that.)
- blyry 6y agoPure speculation, but probably the same reason they ran Oracle for so long -- not many cdn providers when amazon.com launched. Afaik akamai still has one of the biggest edge networks of all the providers.
- Legogris 6y agoI'd wager at least part of it is the same reason many companies use CloudFront: Separate point of failure in case of downtime at the AWS side, some one else shouldering DDoS.
- MaxBarraclough 6y agoI asked the same question 7 days ago: https://news.ycombinator.com/item?id=23537135 https://news.ycombinator.com/item?id=23537135
- 6y ago
- closeparen 6y agoTraceroute’s trick is amusing. It abuses the TTL field, sending out packets with too-low TTLs and waiting to see who complains about them. When layers reveal themselves they are doing it voluntarily, and those wise to the game can choose not to participate, or to troll it. https://www.theregister.com/2013/02/15/star_wars_traceroute/ https://www.theregister.com/2013/02/15/star_wars_traceroute/
- m3047 6y agoThose are IP packet TTLs. Also useful in some cases for diagnosing mysterious RSTs and SYNs. Also useful in some cases to purposefully set low if there is route flapping.
- Tepix 6y agoIt's not abusing the TTL, it's working as designed
- closeparen 6y agoAFAIK the design purpose of TTL is to prevent an infinite loop in case the route contains a cycle.
- allarm 6y agoThe IP TTL field is unrelated to the topic - DNS and IP are sitting on the different OSI layers. Also, the traceroute command does not abuse this field, it’s just the way the IP protocol works. I.e. eBGP sets TTL to 1 by default, to prevent establishing relationships with the neighbors further that one hop away.
- m3047 6y agoMy favorite dig trick is asking caching resolvers with +norecurse to see what's in the cache. Of course some resolvers don't honor it, just like they don't support ANY.
- inamberclad 6y agoI thought DNS was a UDP protocol where you fire off a packet and use the first response that comes back. It's not always one to one.
- eat_veggies 6y agoUDP is stateless but DNS is definitely 1:1 in that there is a notion of a query and its reply. This relationship is made even clearer when you consider DoH.
- JoshMcguigan 6y agoDNS infrastructure is really interesting. I did a bit of a deep dive on it a few months ago, culminating in running my own authoritative name servers [0] for a while. [0]: https://www.joshmcguigan.com/blog/run-your-own-dns-servers/ https://www.joshmcguigan.com/blog/run-your-own-dns-servers/
- rhizome 6y agoOne neat way of retaining that control is running your own SOA(s), but getting robust secondaries and listing those in WHOIS so that they take all of the wild queries. Then you just work with your little SOA and everything just propagates as necessary and you don't get hammered.
- jrockway 6y agoThis reminds me that I wish DNS had some way to define a load balancing algorithm for clients to use, so browsers could make load balancing decisions. This would eliminate the need for virtual IP addresses, having to pass originating subnet information up recursive queries, having to remove faulty VIPs (or hosts) from DNS, etc. It is baffling to me that inside the datacenter, I can control the balancing strategy for every service-to-service transaction, but for the end user's browser, all I can do is some L3 hacks to make two routers appear as one (for failover purposes). L3 balancing would be completely unnecessary if I could just program the user agent to go to the right host, after all. The end result is unnecessary cost and complexity multiplied over a billion websites.
- LinuxBender 6y agoThat is a use of SRV records [1], however it was not accepted into the HTTP protocol specification. I bring it up every time there is a new protocol version but I am too lazy to write an RFC addendum for it and hope that someone else will. Existing protocols may not be modified in this manor once ratified. Maybe HTTP/4.0? /s Some applications use SRV records for load balancing. Many VoIP and video conferencing apps do this. There is a better list on Wikipedia. _service._proto.name. TTL class SRV priority weight port target. [1] - https://en.wikipedia.org/wiki/SRV_record https://en.wikipedia.org/wiki/SRV_record
- jrockway 6y agoYeah, I always liked SRV records. It seems that they proved inadequate for gRPC balancing, so there are new experiments in progress (mostly xDS).
- 1996 6y agoHow would it be better than round robin DNS with low TTL?
- jrockway 6y agoBasically, it affords you the ability to cache for longer and still end up with users able to go to your website. Right now, you can try resolving common hosts, and you will see that they often provide several hosts in response to a lookup. What the browser does with those IPs is up to the browser, the standard does not define what to do. What the administrator that sets up that record wants is "send to whichever one of these seems healthy", and some browsers do do that. Other browsers just pick one at random and report failure, so your redundancy makes the system more likely to break. What I want is a way to define what to do in this case. Maybe you want to try them all in parallel and pick the first to respond (at the TCP connection level). Maybe you want to try them sequentially. Maybe you want to open a connection to all of them and send 1/n requests to each. Right now, there is no way to know what the service intends, so the browser has to guess. And each one guesses differently. (You will notice that people like Google and Cloudflare skillfully respond with only one record with a 5 minute TTL. That is so the behavior of the browser is well defined, but it also eats their entire year of 99.999% uptime with one bad reply. Your systems had better be very reliable if DNS issues can eat a year's worth of error budget.)
- muppetman 6y agoGlad to see this. One of my (stupid) pet peeves is people that say "You have to wait for the DNS to propogate". DNS does not propogate. What you're actually waiting for is the cache TTL to expire so those name-servers that have cached it have to query the real answer again, thus getting the newly pushed information. Of course it appears exactly like it "takes time to propagate" which is why it's actually a pretty sound description of what's happening, and thus why it's a stupid pet peeve. Pointless rant ends.
- rovr138 6y agoThe change is propagating through the network, but it’s not a push like most would assume based on the wording
- HenryBemis 6y agoOld guy here: maybe people were confusing DNS with the WINS service that was helping to propagate ("replicate") the name/servers changes 20 years ago?
- tialaramex 6y agoYes, I'm annoyed about this too. The most egregious case I've seen was an Amiga site. The site went down and for several days reported that users would need to wait for the updated records to propagate and lots of loyal fans were insisting anybody who couldn't read the site was just being too impatient. What was actually wrong? They wrote their new IP address as a DNS name in their DNS configuration rather than as an IP address. Once they fixed that it began working and they acted as though that was just because now it had successfully propagated. On the other hand propagation is a thing when it comes to distributing modified DNS records to multiple notionally authoritative DNS servers. This can be a problem for using Let's Encrypt dns-01 challenges for example, especially with a third party DNS provider. Suppose you write a TXT record to pass dns-01 and get a wildcard certificates for your domain example.com. You submit it to your provider's weird custom API and it says OK. Unfortunately when you do this all it really did was write the updated TXT record to a text file on an SFTP server. Each of the provider's say three authoritative DNS servers (mango, lime, kiwi) check this site every five minutes, download any updated files and begin serving the new answers. Still they said OK, so you call Let's Encrypt and say you're ready to pass the challenge. Let's Encrypt calls authoritative server kiwi, which has never seen this TXT record and you fail the challenge. So you check DNS - your cache infrastructure calls lime, which has updated and gives the correct answer, it seems like everything is fine, so you report a bug with Let's Encrypt. But nothing was wrong on their side. Now, unlike typical "DNS propagation" myths the times for authoritative servers are usually minutes and can be only seconds for a sensible design (SFTP servers is not a sensible design) so you can just add a nice generous allowance of time and it'll usually work. But clearly the Right Thing™ is to have an API that actually confirms the authoritative servers are updated before returning OK.
- asciimike 6y agohttps://howdns.works https://howdns.works is one of my favorite educational booklets on the subject. Not as in depth as many other resources, but highly amusing and fairly sticky.
- logikblok 6y agoThis is brilliant thanks.
- ricardo81 6y agoRecursive DNS servers can also throw you off the scent a bit by giving you an answer that is not the same as the authoritative server. I've seen 8.8.8.8 return something other than NXDOMAIN for some domains that do not exist Cloudflare will not honour dns ANY requests Knowing how to query the authoritative nameservers is a handy tool for debugging.
- LogicX 6y agoAgreed. There's a lot of 'magic' that goes into running a quality recursive resolver, least of which is eDNS0 and EDNS Client Subnet - which intentionally returns different answers based on the requester's source IP -- in most cases for the most-optimal CDN location to be returned. Test with: dig @ns1.google.com www.google.es +subnet=193.8.172.75/24 dig @ns1.google.com www.google.es +subnet=157.88.0.0/16 Note how you get different IPs returned.
- preinheimer 6y agoHere's a pretty clear demo of different results around the world: https://wheresitup.com/demo/results/5ef1403cb8e31e3fb3298503 https://wheresitup.com/demo/results/5ef1403cb8e31e3fb3298503
- r1ch 6y agoAnother cool thing about DNS - you only need the IP of a single root server to be able to get the nameservers, IPs and everything else to resolve any name in any TLD. The hierarchical nature of DNS is very neat to see in action. I built a toy DNS checker tool[0] while I was learning about it to get a more visual overview, and it ended up being one of my tools I still use every few days to verify a domain is properly delegated if I suspect it has issues. [0] https://r-1.ch/r1dns/ https://r-1.ch/r1dns/ (and yes it has a lot of bugs, don't break it :)).
- kalleboo 6y agoOff topic, you wouldn't be the R1ch of old something awful forums fame, would you? I used to run a waffleimages mirror :)
- r1ch 6y agoYup, that's me! The SA forums inspired a lot of my random side projects, the TF2 server integration stuff was especially fun (got me into reverse engineering Valve's game .so to fix bugs). I miss those days when I had all the free time in the world :).
- z3t4 6y agoI have an idea: Because DNS requests are made from a server close to the user, the TLD should use a GEO table in order to give the the two closest DNS servers. Kinda like Anycast without having to configure routing/BGP sessions.
- m3047 6y agoA lot of DNS infrastructure is anycast.
- z3t4 6y agoYes. But most domains eg. websites dont have Anycast. And Anycast is expensive if you just have a private web site or blog. And Anycast services have poor coverage, its only clodflare that has decent coverage. But they only offer proper DNS service to enterprise customers.
- m3047 6y ago> But most domains eg. websites dont have Anycast. Are you talking about the (mathematical) "domain" in the DNS specs, or the popular domain i.e. the web server? The latter is arguably true, in which case the geoip proposition is moot: there is only one web server. Maybe you mean the web server has multiple addresses instead of being anycast. Ok, yes that happens; and some DNS servers do use geoip to tailor replies to try and hand the closest address. Here is from tbe BIND ARM: "By default, if a DNS query includes an EDNS Client Subnet (ECS) option which encodes a non-zero address prefix, then GeoIP ACLs will be matched against that address prefix. Otherwise,they are matched against the source address of the query" Regarding the former, does anyone have info on how many DNS providers use anycast? I think a lot; or maybe I should say that a lot of domains are hosted on anycast, the DNS isn't as distributed as it used to be. If you're using DNS as a distributed key/value store, I hope you're doing a better job thinking about externalities (leakage) than e.g. the antivirus companies in terms of locating authoritatives and how you update them opaquely. Personally I think stub resolvers are stuck in the 1980s. They could do a lot more by monitoring traffic health and editing DNS replies. Due to peering arrangements you could be in the same IX as someone else but that might not be the best route. Traceroute, SYN exchanges, (IP) TTLs might be better signals for determining the health of a particular path. I'd never thought about it until this thread started, maybe the stub resolver could use netflow analysis to inform editing the responses it returns to the applications.
- fomine3 6y agoThe title "update your DNS" looks ambiguous
- rkagerer 6y agoHow distributed is DNS these days in practice, compared to 10 or 20 years ago? If the major internet powers agreed to stop serving responses to DNS servers they detected weren't respecting reasonable TTL's, do you think they could "bully" the industry into tightening things up? (Kind of how Google and others compelled the web toward widespread HTTPS)
- rswail 6y agoI'm not sure why people don't run their own caching server instead of stubs. Or at least for the LAN? You don't have to use your ISPs DNS servers, unless they are evil and capture port 53.
- Tepix 6y agoThere is a very elegant way to update your DNS if you are running djbdns: You can optionally specify date ranges for every record![1] The server will automatically adjust the TTL. By having two records with different time ranges you can switch IP addresses at an exact moment. The timestamps are provided in DJBs TAI64 format, use something like https://github.com/hinnerk/py-tai64 https://github.com/hinnerk/py-tai64 to convert them. [1] config file spec at https://cr.yp.to/djbdns/tinydns-data.html https://cr.yp.to/djbdns/tinydns-data.html
- mrb 6y ago"What I’d expect to happen in practice when updating a DNS record with a 5 minute TTL is that a large percentage of clients will move over to the new IPs quickly (like within 15 minutes)," That's not true. The vast majority of clients will move to the new IP within the TTL, within 5min (not 15min). Then there will be some stragglers that slowly update over the next hours/days (typically poorly written bots) Source : my own experience updating a site with 500k hits per month and sniffing and watching network traffic at the 3 endpoints: DNS, old IP, new IP.
- vbsteven 6y agoOr any proxy using default nginx configuration which caches DNS resolution for upstream blocks at first use and never invalidate until the config is reloaded or Nginx restarted.
- jamesholden 6y agoAs someone newly trying to learn DNS, I don't _use_ 8.8.8.8 personally. So I was confused at first why they kept offering it up on the page. It might help in the first reference to it, so say 'Google DNS' along with the 8.8.8.8
- devdas 6y ago1.1.1.1 is Cloudflare. 8.8.8.8 is Google 9.9.9.9 is PCH/IBM as Quad 9. They all also offer IPv6.