8 ms·

How to Download a List of All Registered Domain Names

170 points by jwcrux 11y ago

jlgaddis 11y ago
Interesting, thanks for the pointer. I've wondered about this previously as I run my own blacklists for $work's mail servers, thinking about how I could slightly "penalize" brand new domain names and such, correlating "spammy" domains with certain nameservers and such.
- dpifke 11y ago
  There's a DNSBL for that: http://support-intelligence.com/dob/ http://support-intelligence.com/dob/
vortico 11y ago
What if someone were to maintain an unofficial list with one domain per line, freely available as a daily torrent or served directly? Would there be a rights problem with mirroring and filtering ICANN data?
- ajcr 11y ago
  Unfortunately, as part of the application you are compelled to sign forms promising that you won't make "significant" parts of the zone file publicly available in any way (at least this was my experience when applying to Verisign for .com and .net zone file access).
  - vortico 11y ago
    I see. Oh well.
- latenightcoding 11y ago
  Same question here
- kyriakos 11y ago
  isn't it essentially public data though?
- radicalbyte 11y ago
  Why not just upload it to Github? Then you only need to send the deltas for every update.
  - r3bl 11y ago
    I was just thinking of creating a bot that will update the list(s) and push the changes back on GitHub every day. But first, I definitely have to see if it's against GitHub's terms of service.
    - radicalbyte 11y ago
      I think that GitHub would love the publicity personally. They strike me as being very savvy regards this sort of thing ;)
      - pyvpx 11y ago
        does GitHub need any publicity? honest question as I've found that anyone who would ever use the functionality GitHub provides is already very aware of git and GitHub.
        allannienhuis 11y ago
        people shift between services like github and bitbucket and alternatives all the time. Perhaps not often on an individual basis, but at any one time many people are deciding where to put their stuff. Almost anything that gets the name of a particular service bumped up to the top of someone's consciousness for a little while will shift some of those decisions toward that service. This is why even the world's most popular brands (Apple, Coke, etc) never stop spending money on marketing / PR :)
- scandinavian 11y ago
  A lot of tlds don't provide zone files unless you are a registrar. They would probably not be happy if someone put those out to the pubic. For com and the likes they would probably not care as much. The best public list of domains I have found is the Project Sonar DNS (ANY) scans. I don't know how they do it, but their scans are pretty complete, at least for.dk domains, which are the ones I use. https://scans.io/study/sonar.fdns https://scans.io/study/sonar.fdns
  - ndr 11y ago
    It's quite slow to download from that site, I guess there are a lot of HN readers consuming their bandwidth :)
    - scandinavian 11y ago
      Their download speeds are often bad sadly, they really should provide a torrent. But it's free and they have some really interesting datasets, so it's worth the wait.
  - ausjke 11y ago
    I started to look at this one week ago, wish it provide some restful api for this useful database
- SmokyBorbon 11y ago
  It could be used to create a competing system. ICANN would never allow this. If someone tried to put this together, I think they would quickly find their access to the data revoked.
zamalek 11y ago
Now someone just needs to train an NN to recognize botnets and spam domains.
- ech 11y ago
  cough https://news.ycombinator.com/item?id=10352001 https://news.ycombinator.com/item?id=10352001 cough we're in the process of getting all the zone files we can to reduce the amount of DNS requests we have to do, but the real kicker is whois databases, for example AFNIC asks for 10K€ to get access to a copy of the database...
  - zamalek 11y ago
    Oh I missed that. You're doing good work there!
ps4fanboy 11y ago
I think its sad how closed this data is.
- talideon 11y ago
  There are very good reasons for this data being closed, not least of which is that allowing zone transfer by arbitrary individuals is an excellent way of allowing your DNS server to be DOS'd.
  - ps4fanboy 11y ago
    How is that achieved with read only access to the list of registered domains?
    - talideon 11y ago
      I'm not sure what you mean. Do you know what a zone transfer is? If you wanted to get a list of the domains and records published in a nameserver, you would perform a zone transfer. Because that can amount to quite a bit of information being transferred, if a nameserver allows unrestricted zone transfers, that's a vector for a denial of service attack against that nameserver. If you're a domain registry, your zone files are huge. Allowing arbitrary zone transfers could potentially put massive sustained strain on their DNS infrastructure. And thus because only a very small number of nameservers really need to be able to perform zone transfers against their nameservers, they're better off locking down the ability. If you're running your own nameservers, then it's still worth locking down zone transfers for similar reasons. At the very least, it gives you a degree of defence in depth as you're giving attackers less of an opportunity to gather information on the structure of your network. If they could simply do a zone transfer to find out all the names in a given zone, then they don't have to do more costly brute force enumeration to guess at the hosts in the zone. Take a read of this for why, if you run your own nameservers, you shouldn't allow arbitrary zone transfers: http://www.iodigitalsec.com/dns-zone-transfer-axfr-vulnerability/ http://www.iodigitalsec.com/dns-zone-transfer-axfr-vulnerabi...
jtwaleson 11y ago
A couple of months ago I processed all metadata from the Common Crawl project for all indexed domain names. This was about 10TB of metadata and resulted in 26 million domain names. EC2 costs were only about 10$ to process this. If anyone is interested, let me know. edit: available as torrent here: https://all-certificates.s3.amazonaws.com/domainnames.gz?torrent https://all-certificates.s3.amazonaws.com/domainnames.gz?tor...
- mortenlarsen 11y ago
  Please make a torrent.
  - jtwaleson 11y ago
    https://all-certificates.s3.amazonaws.com/domainnames.gz?torrent https://all-certificates.s3.amazonaws.com/domainnames.gz?tor...
    - pibefision 11y ago
      thank you for this!
      - jtwaleson 11y ago
        just out of curiosity, what are you (and the other commenters) planning to use this for? I myself was trying to download all X509 server certificates on the Internet I could find to: - gain some insight into the dealings of CAs - see if private keys were reused across multiple sites - check the average expiry date of certificates etc.
        flexd 11y ago
        My intentions is that I want a complete list of .no (Norwegian) domains, and http://norid.no http://norid.no refuse to give out that list to anyone. I would like to be able to continuously check the Norwegian IP-space for compromised sites, because it would be interesting to see. Of course doing this on a bigger scale would be cool as well.
nly 11y ago
A warning about parsing zone files... the grammar is deceptively tricky. While TLD registries will probably provide you with files in a sane subset[0] of that specified in RFC 1035, there are a number of things that will NOT work in general: - Splitting the file in to lines (paren-blocks and quoted strings can span lines, strings can contain ';' etc). - Splitting the file on whitespace (it's significant in column 1 and inside strings) - Applying a regex (you'll need lookahead for conditional matching and it'll get ugly fast) Don't go down the road of assuming it's a simple delimited file. A few references: https://www.nlnetlabs.nl/projects/nsd/documentation.html https://www.nlnetlabs.nl/projects/nsd/documentation.html http://www.verycomputer.com/96_5ad11cc47053d8b0_1.htm http://www.verycomputer.com/96_5ad11cc47053d8b0_1.htm [0] See page 9 of https://archive.icann.org/en/topics/new-gtlds/zfa-strategy-paper-12may10-en.pdf https://archive.icann.org/en/topics/new-gtlds/zfa-strategy-p...
- fanf2 11y ago
  Yep. I use the BIND tool named-compilezone to canonicalize zone files, which allows me to apply simple regex parsing, because I can assume one record per line, all fields present, and no abbreviated names. Main disadvantage is it is not very fast.
  - nly 11y ago
    It's probably slow because by default it does a bunch of DNS queries to foreign zones to see if your NS records etc are good.
mike-cardwell 11y ago
What about CCTLDs?
- ech 11y ago
  ccTLDs are the responsibility of "designated managers" for each sovereign country (see https://tools.ietf.org/html/rfc1591 https://tools.ietf.org/html/rfc1591).
- sneg55 11y ago
  They available from a third party source such as http://domains-index.com/ http://domains-index.com/
  - mike-cardwell 11y ago
    Yeah. My point was, he wrote an article titled "How to Download a List of All Registered Domain Names", and then didn't even mention the existence of cctlds. Which is like writing an article titled "How to learn how to speak every language", and then pretending there are only 10 languages in existence.
irfan 11y ago
I had been downloading the zone file for .PK domains on daily bases until they blocked the zone transfers. Based on comparison of these daily zone files I managed to publish the statistics [1] and also broke the news about hacked .PK domains [2] which was picked up by all leading tech blogs and news agencies. Currently, I cannot find a way to get the zone file even by officially requesting the registry manager. [1]: https://www.i.com.pk/pknic-domain-registration-statistics/ https://www.i.com.pk/pknic-domain-registration-statistics/ [2]: https://www.i.com.pk/110-pk-domains-managed-by-markmonitor-got-hacked-by-turkish-hackers/ https://www.i.com.pk/110-pk-domains-managed-by-markmonitor-g...
axaxs 11y ago
FWIW, a TLD zone file does not contain every registered domain name, just those with DNS records. There is typically a good amount of domain names registered but without records, for reasons such as reserved names, malicious content takedowns, etc.
- larrys 11y ago
  Exactly. The title "How to Download a List of All Registered Domain Names" is not correct. For example in .com .net if a registrar puts a domain "on hold" that pulls it from the zone file. As such it will not appear in any zone file download. This is in fact a way for a company to keep a domain under wraps. As such: register domain name (in .com ) pull the name servers The name won't be in any zone file. And will be off the radar until it has nameservers or until someone guesses to see if it's been registered. (Note other registries work differently (or the same) the above is specific to .com and .net)
canow 11y ago
would it be easier to download a list of available domains?
- talideon 11y ago
  No. The list of available domains is the list of all possible domains less the list of registered domains. The list of registered domains is vastly smaller than the list of possible domains. The list of available domains would mostly consist of junk nobody would be interested in.
deleted 11y ago
[deleted]
ben_utzer 11y ago
This list would be useful for my attempt for a list of parked/squatted domains..

170 points by jwcrux 11y ago

Interesting, thanks for the pointer. I've wondered about this previously as I run my own blacklists for $work's mail servers, thinking about how I could slightly "penalize" brand new domain names and such, correlating "spammy" domains with certain nameservers and such.

dpifke 11y ago
There's a DNSBL for that: http://support-intelligence.com/dob/ http://support-intelligence.com/dob/

vortico 11y ago

What if someone were to maintain an unofficial list with one domain per line, freely available as a daily torrent or served directly? Would there be a rights problem with mirroring and filtering ICANN data?

ajcr 11y ago
Unfortunately, as part of the application you are compelled to sign forms promising that you won't make "significant" parts of the zone file publicly available in any way (at least this was my experience when applying to Verisign for .com and .net zone file access).
- vortico 11y ago
  I see. Oh well.
latenightcoding 11y ago
Same question here
kyriakos 11y ago
isn't it essentially public data though?
radicalbyte 11y ago
Why not just upload it to Github? Then you only need to send the deltas for every update.
- r3bl 11y ago
  I was just thinking of creating a bot that will update the list(s) and push the changes back on GitHub every day. But first, I definitely have to see if it's against GitHub's terms of service.
  - radicalbyte 11y ago
    I think that GitHub would love the publicity personally. They strike me as being very savvy regards this sort of thing ;)
    - pyvpx 11y ago
      does GitHub need any publicity? honest question as I've found that anyone who would ever use the functionality GitHub provides is already very aware of git and GitHub.
      - allannienhuis 11y ago
        people shift between services like github and bitbucket and alternatives all the time. Perhaps not often on an individual basis, but at any one time many people are deciding where to put their stuff. Almost anything that gets the name of a particular service bumped up to the top of someone's consciousness for a little while will shift some of those decisions toward that service. This is why even the world's most popular brands (Apple, Coke, etc) never stop spending money on marketing / PR :)
scandinavian 11y ago
A lot of tlds don't provide zone files unless you are a registrar. They would probably not be happy if someone put those out to the pubic. For com and the likes they would probably not care as much. The best public list of domains I have found is the Project Sonar DNS (ANY) scans. I don't know how they do it, but their scans are pretty complete, at least for.dk domains, which are the ones I use. https://scans.io/study/sonar.fdns https://scans.io/study/sonar.fdns
- ndr 11y ago
  It's quite slow to download from that site, I guess there are a lot of HN readers consuming their bandwidth :)
  - scandinavian 11y ago
    Their download speeds are often bad sadly, they really should provide a torrent. But it's free and they have some really interesting datasets, so it's worth the wait.
- ausjke 11y ago
  I started to look at this one week ago, wish it provide some restful api for this useful database
SmokyBorbon 11y ago
It could be used to create a competing system. ICANN would never allow this. If someone tried to put this together, I think they would quickly find their access to the data revoked.

zamalek 11y ago

Now someone just needs to train an NN to recognize botnets and spam domains.

ech 11y ago
cough https://news.ycombinator.com/item?id=10352001 https://news.ycombinator.com/item?id=10352001 cough we're in the process of getting all the zone files we can to reduce the amount of DNS requests we have to do, but the real kicker is whois databases, for example AFNIC asks for 10K€ to get access to a copy of the database...
- zamalek 11y ago
  Oh I missed that. You're doing good work there!

ps4fanboy 11y ago

I think its sad how closed this data is.

talideon 11y ago
There are very good reasons for this data being closed, not least of which is that allowing zone transfer by arbitrary individuals is an excellent way of allowing your DNS server to be DOS'd.
- ps4fanboy 11y ago
  How is that achieved with read only access to the list of registered domains?
  - talideon 11y ago
    I'm not sure what you mean. Do you know what a zone transfer is? If you wanted to get a list of the domains and records published in a nameserver, you would perform a zone transfer. Because that can amount to quite a bit of information being transferred, if a nameserver allows unrestricted zone transfers, that's a vector for a denial of service attack against that nameserver. If you're a domain registry, your zone files are huge. Allowing arbitrary zone transfers could potentially put massive sustained strain on their DNS infrastructure. And thus because only a very small number of nameservers really need to be able to perform zone transfers against their nameservers, they're better off locking down the ability. If you're running your own nameservers, then it's still worth locking down zone transfers for similar reasons. At the very least, it gives you a degree of defence in depth as you're giving attackers less of an opportunity to gather information on the structure of your network. If they could simply do a zone transfer to find out all the names in a given zone, then they don't have to do more costly brute force enumeration to guess at the hosts in the zone. Take a read of this for why, if you run your own nameservers, you shouldn't allow arbitrary zone transfers: http://www.iodigitalsec.com/dns-zone-transfer-axfr-vulnerability/ http://www.iodigitalsec.com/dns-zone-transfer-axfr-vulnerabi...

jtwaleson 11y ago

A couple of months ago I processed all metadata from the Common Crawl project for all indexed domain names. This was about 10TB of metadata and resulted in 26 million domain names. EC2 costs were only about 10$ to process this. If anyone is interested, let me know. edit: available as torrent here: https://all-certificates.s3.amazonaws.com/domainnames.gz?torrent https://all-certificates.s3.amazonaws.com/domainnames.gz?tor...

mortenlarsen 11y ago
Please make a torrent.
- jtwaleson 11y ago
  https://all-certificates.s3.amazonaws.com/domainnames.gz?torrent https://all-certificates.s3.amazonaws.com/domainnames.gz?tor...
  - pibefision 11y ago
    thank you for this!
    - jtwaleson 11y ago
      just out of curiosity, what are you (and the other commenters) planning to use this for? I myself was trying to download all X509 server certificates on the Internet I could find to: - gain some insight into the dealings of CAs - see if private keys were reused across multiple sites - check the average expiry date of certificates etc.
      - flexd 11y ago
        My intentions is that I want a complete list of .no (Norwegian) domains, and http://norid.no http://norid.no refuse to give out that list to anyone. I would like to be able to continuously check the Norwegian IP-space for compromised sites, because it would be interesting to see. Of course doing this on a bigger scale would be cool as well.

nly 11y ago

A warning about parsing zone files... the grammar is deceptively tricky. While TLD registries will probably provide you with files in a sane subset[0] of that specified in RFC 1035, there are a number of things that will NOT work in general: - Splitting the file in to lines (paren-blocks and quoted strings can span lines, strings can contain ';' etc). - Splitting the file on whitespace (it's significant in column 1 and inside strings) - Applying a regex (you'll need lookahead for conditional matching and it'll get ugly fast) Don't go down the road of assuming it's a simple delimited file. A few references: https://www.nlnetlabs.nl/projects/nsd/documentation.html https://www.nlnetlabs.nl/projects/nsd/documentation.html http://www.verycomputer.com/96_5ad11cc47053d8b0_1.htm http://www.verycomputer.com/96_5ad11cc47053d8b0_1.htm [0] See page 9 of https://archive.icann.org/en/topics/new-gtlds/zfa-strategy-paper-12may10-en.pdf https://archive.icann.org/en/topics/new-gtlds/zfa-strategy-p...

fanf2 11y ago
Yep. I use the BIND tool named-compilezone to canonicalize zone files, which allows me to apply simple regex parsing, because I can assume one record per line, all fields present, and no abbreviated names. Main disadvantage is it is not very fast.
- nly 11y ago
  It's probably slow because by default it does a bunch of DNS queries to foreign zones to see if your NS records etc are good.

mike-cardwell 11y ago

What about CCTLDs?

ech 11y ago
ccTLDs are the responsibility of "designated managers" for each sovereign country (see https://tools.ietf.org/html/rfc1591 https://tools.ietf.org/html/rfc1591).
sneg55 11y ago
They available from a third party source such as http://domains-index.com/ http://domains-index.com/
- mike-cardwell 11y ago
  Yeah. My point was, he wrote an article titled "How to Download a List of All Registered Domain Names", and then didn't even mention the existence of cctlds. Which is like writing an article titled "How to learn how to speak every language", and then pretending there are only 10 languages in existence.

irfan 11y ago

I had been downloading the zone file for .PK domains on daily bases until they blocked the zone transfers. Based on comparison of these daily zone files I managed to publish the statistics [1] and also broke the news about hacked .PK domains [2] which was picked up by all leading tech blogs and news agencies. Currently, I cannot find a way to get the zone file even by officially requesting the registry manager. [1]: https://www.i.com.pk/pknic-domain-registration-statistics/ https://www.i.com.pk/pknic-domain-registration-statistics/ [2]: https://www.i.com.pk/110-pk-domains-managed-by-markmonitor-got-hacked-by-turkish-hackers/ https://www.i.com.pk/110-pk-domains-managed-by-markmonitor-g...

axaxs 11y ago

FWIW, a TLD zone file does not contain every registered domain name, just those with DNS records. There is typically a good amount of domain names registered but without records, for reasons such as reserved names, malicious content takedowns, etc.

larrys 11y ago
Exactly. The title "How to Download a List of All Registered Domain Names" is not correct. For example in .com .net if a registrar puts a domain "on hold" that pulls it from the zone file. As such it will not appear in any zone file download. This is in fact a way for a company to keep a domain under wraps. As such: register domain name (in .com ) pull the name servers The name won't be in any zone file. And will be off the radar until it has nameservers or until someone guesses to see if it's been registered. (Note other registries work differently (or the same) the above is specific to .com and .net)

canow 11y ago

would it be easier to download a list of available domains?

talideon 11y ago
No. The list of available domains is the list of all possible domains less the list of registered domains. The list of registered domains is vastly smaller than the list of possible domains. The list of available domains would mostly consist of junk nobody would be interested in.

deleted 11y ago

[deleted]

ben_utzer 11y ago

This list would be useful for my attempt for a list of parked/squatted domains..