3 ms·

Does anyone have a link to a blog that explains why I can't have every DNS entry in a database at home? I don't want a 'you could never get this 1%" because of

Does anyone have a link to a blog that explains why I can't have every DNS entry in a database at home? I don't want a 'you could never get this 1%" because of X straight up. What percentage could I get and why is this not easy?

djbusby 2y ago
Well, the database might be large. Millions of domain names, each with like a dozen (or more) records. (maybe 1k * 30 million domains). But mostly it's about how the DNS changes, a lot. Sites move, geo records, etc. Getting those changes to every household would be a lot of traffic. Before DNS folk were sharing a common /etc/hosts file - and quickly determined there had to be a better way. So, DNS with a dozen root servers that stay fresh, caching down stream.
- ranger_danger 2y ago
  Also due to Anycast, DNS servers can/do respond with different addresses depending on who requests it and from where.
- tlb 2y ago
  Large for the old days, but a billion names is no longer a big file compared to a AAA game.
- fragmede 2y ago
  A 30 GiB file is nothing these days.
thedougd 2y ago
1) It's distributed. There's not a single database with all the DNS records in existence. Each zone is responsible for hosting and serving their own records. 2) The TTL for a DNS record can be 0. Edit: 3) AXFR is blocked on 'most' zones. You would need this to know all the records in a zone.
- Avamander 2y ago
  Zero TTL is insane though and most caching resolvers allow enforcing a minimum.
  - ectospheno 2y ago
    Last year I started bumping up min ttl on dns caches a bit each week. Stopped at one hour. No one ever complained.
georgyo 2y ago
There are too many reasons to list. But the primary reasons DNS is a hierarchy of DNS services hosted by many distinct groups. And DNS records are queriable publicly, but also consider private. Almost all servers in the world disable AFRX for a variety reasons. Beyond that DNS servers are not just a "database" of records, they are services that can return different results depending on who queried. And this is quite common with CDNs. There are many more reasons, but let's talk about what you could do. You could download a list of all registered domains, and then query all of them for the most common records. It would take hours, not include 99.9 percent all records, and be out of date the second it completes. With this database, you could visit some websites, but many host services on subdomains and you have no way to dynamically get that list when you're populating the database.
Bjartr 2y ago
How would you update that database? Who would you ask for the updates? IPs change out from under domains constantly. How would knowledge of those changes make their way from the owner of the domain to your local database? If you'd want a single source of truth to receive and distribute that info, that's a single point of failure and comes with lots of technical challenges. And it's less "you could never get this 1%" and more "you will never need 99.9% of this" ------------------------------------------------ However, that's all entirely unrelated to this Cloudflare incident, which involved no domains at all, only raw IP addresses. This was an incident involving BGP, which is how the various providers of internet backbone know which other provider they connect to is able to answer requests made to a given IP address, and how changes to that information propagate.
myrmidon 2y ago
Simply because those entries are not static. When Jane Deer creates a new Blog jane-deer.com, your local database would not know which server to contact to resolve it, and nobody is gonna notify the database in your home. So you would have to contact some other kind of database outside your home to find out how to resolve that domain--which you might have already realized sounds a lot like a (non-local) DNS server...
chabad360 2y ago
The gist is because DNS gets updated. Your computer already has a local cache of all DNS responses it receives and relies on those until that entry expires (as set by the TTL field). If you want to run a local resolver, you can, but it wouldn't solve this problem, because you'll still need to update your cache regularly and guess how you do that? With DNS. The next question from there might be why do DNS responses change so frequently? I don't have an exact statistic, but most of them don't (I would guess 90%), so you could probably indefinitely cache those. But the ones that do, iirc, it's usually to allow quicker changes (esp. if the site is under active development), load balancing, global distribution, etc. I hope that's helpful.
DanAtC 2y ago
As others have said, DNS is distributed/hierarchical/delegated. But if you wanted to try... First, grab the root zone file which points to all the TLD nameservers: https://www.internic.net/domain/root.zone https://www.internic.net/domain/root.zone Next, get zone files for the TLDs themselves. Not every TLD makes that freely or easily available, but https://github.com/jschauma/tld-zoneinfo https://github.com/jschauma/tld-zoneinfo is an excellent resource. The author of the above repo runs https://netmeister.org/tldstats/ https://netmeister.org/tldstats/ which will give you an idea of scale; they're able to get 98.89% of all TLDs. (Correction: zone files for ccTLDs are mostly unavailable. That 98.89% is only domain count coverage, not zone file coverage.)
- Avamander 2y ago
  I guess this alone might speed things up a bit, if cached, especially with DNSSEC. I wonder if anyone has attempted it.
patmorgan23 2y ago
Because you don't own the data and it changes. Also lots of people do DNS bases load balancing.

djbusby 2y ago

Well, the database might be large. Millions of domain names, each with like a dozen (or more) records. (maybe 1k * 30 million domains). But mostly it's about how the DNS changes, a lot. Sites move, geo records, etc. Getting those changes to every household would be a lot of traffic. Before DNS folk were sharing a common /etc/hosts file - and quickly determined there had to be a better way. So, DNS with a dozen root servers that stay fresh, caching down stream.

ranger_danger 2y ago
Also due to Anycast, DNS servers can/do respond with different addresses depending on who requests it and from where.
tlb 2y ago
Large for the old days, but a billion names is no longer a big file compared to a AAA game.
fragmede 2y ago
A 30 GiB file is nothing these days.

thedougd 2y ago

1) It's distributed. There's not a single database with all the DNS records in existence. Each zone is responsible for hosting and serving their own records. 2) The TTL for a DNS record can be 0. Edit: 3) AXFR is blocked on 'most' zones. You would need this to know all the records in a zone.

Avamander 2y ago
Zero TTL is insane though and most caching resolvers allow enforcing a minimum.
- ectospheno 2y ago
  Last year I started bumping up min ttl on dns caches a bit each week. Stopped at one hour. No one ever complained.

georgyo 2y ago

There are too many reasons to list. But the primary reasons DNS is a hierarchy of DNS services hosted by many distinct groups. And DNS records are queriable publicly, but also consider private. Almost all servers in the world disable AFRX for a variety reasons. Beyond that DNS servers are not just a "database" of records, they are services that can return different results depending on who queried. And this is quite common with CDNs. There are many more reasons, but let's talk about what you could do. You could download a list of all registered domains, and then query all of them for the most common records. It would take hours, not include 99.9 percent all records, and be out of date the second it completes. With this database, you could visit some websites, but many host services on subdomains and you have no way to dynamically get that list when you're populating the database.

Bjartr 2y ago

How would you update that database? Who would you ask for the updates? IPs change out from under domains constantly. How would knowledge of those changes make their way from the owner of the domain to your local database? If you'd want a single source of truth to receive and distribute that info, that's a single point of failure and comes with lots of technical challenges. And it's less "you could never get this 1%" and more "you will never need 99.9% of this" ------------------------------------------------ However, that's all entirely unrelated to this Cloudflare incident, which involved no domains at all, only raw IP addresses. This was an incident involving BGP, which is how the various providers of internet backbone know which other provider they connect to is able to answer requests made to a given IP address, and how changes to that information propagate.

myrmidon 2y ago

Simply because those entries are not static. When Jane Deer creates a new Blog jane-deer.com, your local database would not know which server to contact to resolve it, and nobody is gonna notify the database in your home. So you would have to contact some other kind of database outside your home to find out how to resolve that domain--which you might have already realized sounds a lot like a (non-local) DNS server...

chabad360 2y ago

The gist is because DNS gets updated. Your computer already has a local cache of all DNS responses it receives and relies on those until that entry expires (as set by the TTL field). If you want to run a local resolver, you can, but it wouldn't solve this problem, because you'll still need to update your cache regularly and guess how you do that? With DNS. The next question from there might be why do DNS responses change so frequently? I don't have an exact statistic, but most of them don't (I would guess 90%), so you could probably indefinitely cache those. But the ones that do, iirc, it's usually to allow quicker changes (esp. if the site is under active development), load balancing, global distribution, etc. I hope that's helpful.

DanAtC 2y ago

As others have said, DNS is distributed/hierarchical/delegated. But if you wanted to try... First, grab the root zone file which points to all the TLD nameservers: https://www.internic.net/domain/root.zone https://www.internic.net/domain/root.zone Next, get zone files for the TLDs themselves. Not every TLD makes that freely or easily available, but https://github.com/jschauma/tld-zoneinfo https://github.com/jschauma/tld-zoneinfo is an excellent resource. The author of the above repo runs https://netmeister.org/tldstats/ https://netmeister.org/tldstats/ which will give you an idea of scale; they're able to get 98.89% of all TLDs. (Correction: zone files for ccTLDs are mostly unavailable. That 98.89% is only domain count coverage, not zone file coverage.)

Avamander 2y ago
I guess this alone might speed things up a bit, if cached, especially with DNSSEC. I wonder if anyone has attempted it.

patmorgan23 2y ago

Because you don't own the data and it changes. Also lots of people do DNS bases load balancing.