3 ms·

Is there a practical way to enumerate all the registered internet domains? EG by asking DNS servers for all the domains they know about, and repeating over all

Is there a practical way to enumerate all the registered internet domains? EG by asking DNS servers for all the domains they know about, and repeating over all DNS servers they know about? EDIT: apparently, "asking DNS servers for all the domains they know about" is not something you can really do anymore for security reasons. Guess that idea won't fly lol

onion2k 6mo ago
You could probably extract a lot from https://commoncrawl.org/ https://commoncrawl.org/
- ks2048 6mo ago
  Here's the latest (although, it looks truncated at those having > 1M pages), https://commoncrawl.github.io/cc-crawl-statistics/plots/tld/latestcrawl.html https://commoncrawl.github.io/cc-crawl-statistics/plots/tld/...
  - ccgreg 6mo ago
    The complete list hides in the web graph: https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2026-jan-feb-mar/index.html https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main... and the specific file that's every host we've seen in the latest 3 crawls is: https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2026-jan-feb-mar/host/cc-main-2026-jan-feb-mar-host-ranks.txt.gz https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main...
0x0 6mo ago
Enumeration of the entire DNS space is not available in general, but it does appear that some TLDs offer complete zone files for legitimate research purposes, see for example https://czds.icann.org/help#zone-files https://czds.icann.org/help#zone-files
gucci-on-fleek 6mo ago
> apparently, "asking DNS servers for all the domains they know about" is not something you can really do anymore for security reasons. Guess that idea won't fly lol There are actually a few nameservers that will just give all their domains to anyone who asks [0], but they are very much in the minority. [0]: https://github.com/acidvegas/mdaxfr#tlds-that-allow-axfr https://github.com/acidvegas/mdaxfr#tlds-that-allow-axfr
ks2048 6mo ago
Check out the "Root Zone Database", https://www.iana.org/domains/root/db https://www.iana.org/domains/root/db (this doesn't have .yu)
bit1993 6mo ago
This https://www.whoisxmlapi.com/whois-database-download.php https://www.whoisxmlapi.com/whois-database-download.php is a start, but there's new ones every second so you're going to need to update a lot.
basilikum 6mo ago
certificate transparency logs will give you a lot of that data.

onion2k 6mo ago

You could probably extract a lot from https://commoncrawl.org/ https://commoncrawl.org/

ks2048 6mo ago
Here's the latest (although, it looks truncated at those having > 1M pages), https://commoncrawl.github.io/cc-crawl-statistics/plots/tld/latestcrawl.html https://commoncrawl.github.io/cc-crawl-statistics/plots/tld/...
- ccgreg 6mo ago
  The complete list hides in the web graph: https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2026-jan-feb-mar/index.html https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main... and the specific file that's every host we've seen in the latest 3 crawls is: https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2026-jan-feb-mar/host/cc-main-2026-jan-feb-mar-host-ranks.txt.gz https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main...

0x0 6mo ago

Enumeration of the entire DNS space is not available in general, but it does appear that some TLDs offer complete zone files for legitimate research purposes, see for example https://czds.icann.org/help#zone-files https://czds.icann.org/help#zone-files

gucci-on-fleek 6mo ago

> apparently, "asking DNS servers for all the domains they know about" is not something you can really do anymore for security reasons. Guess that idea won't fly lol There are actually a few nameservers that will just give all their domains to anyone who asks [0], but they are very much in the minority. [0]: https://github.com/acidvegas/mdaxfr#tlds-that-allow-axfr https://github.com/acidvegas/mdaxfr#tlds-that-allow-axfr

ks2048 6mo ago

Check out the "Root Zone Database", https://www.iana.org/domains/root/db https://www.iana.org/domains/root/db (this doesn't have .yu)

bit1993 6mo ago

This https://www.whoisxmlapi.com/whois-database-download.php https://www.whoisxmlapi.com/whois-database-download.php is a start, but there's new ones every second so you're going to need to update a lot.

basilikum 6mo ago

certificate transparency logs will give you a lot of that data.