3 ms·
"I have computers today with enough storage space to hold entire multi-GB public zone files. The storage capability keeps increasing. However I only use a small
by 1vuio0pswjnm7 4y ago
"I have computers today with enough storage space to hold entire multi-GB public zone files. The storage capability keeps increasing. However I only use a small fraction of that data."
What this means is that I do not need to store entire zone files. I only need to store the data for the domain names I will use. The point about storage capability is that this is no longer a limiting factor. When I started using the www, storage space was a limiting factor. I could not store the DNS data for every name I would ever use on a personal computer. Even the RAM on today's computers can be larger than the size of HDDs from the time when I started using the www. Everything has changed.
"For example, I am able to use the www quite effectively for information retrieval (not commerce) without using auto-fetching.^3 I treat www pages as "simple" ones with only one significant component and none controlled by third parties."
What this means is that the set of names I will use is (generally) deterministic. For example, if I aim to access the index.html page at https://example.com https://example.com, I only retrieve the DNS data for example.com. The set of names for which I must retrieve DNS data is known, a priori.^1 To give a more practical example, I start with a list of all the domain names represented in HN submissions (cf. comments). I retrieve DNS data for those names only. (NB. A small minority of www sites submitted to HN do change hosting providers occassionally or change IP addresses relatively frequently.)
Thus when I read HN submissions, I am not performing any remote DNS queries. At an earlier point, I have performed bulk DNS data retrieval for all domain names in HN submmissions. The DNS data is stored in the memory of a localhost forward proxy or in custom zone files served by a localhost authoritative nameserver.
Another example might be domains found in Google Scholar search results. I collect these names from a series of searches then retrieve the DNS data in bulk. Then I can search and retrieve papers from many sources found through Scholar without making remote DNS queries.
There are a variety of sources for bulk DNS data. Some potential sources are
Public zone file access programs (Contact the registry. Many zones are available through ICANN's CZDS program.) https://czds.icann.org https://czds.icann.org
Public scan data (Sadly, Rapid7 recently stopped publishing their foward DNS data.)
DoH open resolvers (Using HTTP/1.1 pipelining.)
Common Crawl archives (By extracting WARC-TARGET-IP.)
1. In contrast to using browser auto-fetching where I have no idea what other domain names might be automatically looked up when I visit example.com.