10 ms·

A penetration tester’s guide to sub-domain enumeration

308 points by adamnemecek 9y ago

polote 9y ago
Great list ! Also you can look at the SSL certificate of a page of the doamin and see if many domains appears inside it. And currently if the website uses letsencrypt and need wildcard DNS, you may find hundreds of domains. Fortunately Letsencrypt wildcard certificates come in January
- nsgi 9y ago
  If the website is using Let's Encrypt then any subdomains using it will be in certificate transparency logs anyway (until it supports wildcard certificates).
  - nawtacawp 9y ago
    Even if they were registered/requested separately? For example, I have mail.domain.com as a separate cert than my domain.com both with letsencrypt.
    - kdbg 9y ago
      Yes, of course. The idea of certificate transparency logs is to be able to audit all of the certificates issued by a CA to detect mistakenly or maliciously issued certificates. You can go ahead and search the logs yourself through google. https://transparencyreport.google.com/https/certificates https://transparencyreport.google.com/https/certificates
  - throwaway613834 9y ago
    Do you know how to find out which certificate issuers DON'T submit certs to transparency logs? Is there a list somewhere?
    - tialaramex 9y ago
      Not logging certificates will eventually cause them not to be trusted in Chrome. The deadline was pushed back but it hasn't done away. Firefox doesn't have a firm intent to do the same but it's developing all the technology. So this is probably not a good idea. Even where a subscriber specifically didn't want their certs logged and the CA doesn't currently log everything if there's ever a problem it'll get logged. This is because both Google and Mozilla will ask to see all the affected certificates, and logging them is the easiest way to satisfy this requirement. If a CA seems to be trying to dodge this usually participants will take whatever form the CA did provide and upload it to the logs. Google's logs don't care who gives them a cert, they can independently determine that it's authentic so they'll log it.
      - throwaway613834 9y ago
        Yeah, but I feel there's at least like a 30%+ chance that their attempt to make this mandatory will stop in its tracks. I just don't see corporations being okay with it.
        jlgaddis 9y ago
        There's some work (in the CA/B?) to allow "redacting" certain information from CT logs. I don't recall all the details but I do remember reading an e-mail thread (maybe linked from HN?) about it in just the last week or so.
        tialaramex 9y ago
        Mandatory logging is a necessary element of the CT design. What we have today is basically like the trial demo of the game. Full CT requires the completion of "gossip protocols", log auditor third parties which detect inconsistencies and a bunch of other stuff that exists today only in the laboratory, you haven't seen the half of it. The option left open by the Chrome team was what's called redaction. The logged document would be partly redacted by for example removing host names. It's not impossible that redaction will happen and end up being accepted, but its biggest proponent was Symantec, who are exiting the business after destroying trust by not exercising proper oversight on their relationship with Korea's CrossCert. And frankly the redaction proposed was getting ludicrous anyway. The trouble is that people who most want this, as with Brass Plate companies, aren't just looking to avoid excess publicity. What they want is to act entirely with impunity. And that's exactly what CT is there to prevent.
porfirium 9y ago
A small question. For a CloudFlare-protected website, I see this in crt.sh: >X509v3 Subject Alternative Name: >DNS Name:sniXXXXXXX.cloudflaressl.com (X are numbers) >DNS Name:(lots of domains, including the one I was researching) >... Does that mean all those domains belong to the same user/person?
- breakingcups 9y ago
  No, Cloudflare batches together many unrelated domains from different customers into a single SSL certificate.
  - programd 9y ago
    Which leads to amusing issues like the following: https://arstechnica.com/information-technology/2015/03/ted-cruz-for-presidents-ssl-certificate-nigerian-prince-headache/ https://arstechnica.com/information-technology/2015/03/ted-c... Though to be fair, this incident is now fairly infamous inside Cloudflare and these days they have ways to deal with it.
dzhiurgis 9y ago
WolframAlpha can display subdomains, not sure how and the format is not easy to use. But it works. How one would one go about finding A domains pointing to specific AS. The target web app in question allows either using their subdomain or pointing to your own. Any ideas how to find those?
jcims 9y ago
The CommonCrawl dataset is another good source of host and subdomain information. The index server is here: http://index.commoncrawl.org http://index.commoncrawl.org There's a client referenced on that page. [ec2-user@x cdx-index-client]$ ./cdx-index-client.py -c CC-MAIN-2017-43 '*.ycombinator.com' 2017-11-11 22:50:08,381: [INFO]: Getting Index From http://index.commoncrawl.org/CC-MAIN-2017-43-index 2017-11-11 22:50:08,387: [INFO]: Starting new HTTP connection (1): index.commoncrawl.org 2017-11-11 22:50:08,468: [INFO]: Fetching 2 pages of *.ycombinator.com 2017-11-11 22:50:08,477: [INFO]: Starting new HTTP connection (1): index.commoncrawl.org 2017-11-11 22:50:08,478: [INFO]: Starting new HTTP connection (1): index.commoncrawl.org 2017-11-11 22:50:10,409: [INFO]: 1 page(s) of 2 finished 2017-11-11 22:50:11,711: [INFO]: 2 page(s) of 2 finished [ec2-user@x cdx-index-client]$ awk -F')' '{print $1}' domain-y* | sort | uniq -c 66 com,ycombinator 38 com,ycombinator,apply 500 com,ycombinator,blog 2 com,ycombinator,fellowship 2 com,ycombinator,macro 24036 com,ycombinator,news 25 com,ycombinator,old 2 com,ycombinator,ycblog
belorn 9y ago
For those on the defending side, remember that the purpose of DNS is to be public. Relying on DNS information to stay secret in order to protect vulnerable application is a bad idea. A service with a SQL injection is unlikely to last long, and as the article shows, it is not hard to enumerate DNS records if you are under a targeted attack.
- feelin_googley 9y ago
  "... remember that the purpose of DNS is to be public." DNS records, e.g. zone files, are supposed to be public information. As such, ICANN requires gTLD registries to provide users with access to them.[FN1] However, various ccTLDs at various times refused to comply, and most still try to pretend their zone files are private. Obviously, as the blog post shows, users can still get a large quantity of the data freely from other sources. Google search results for ccTLD zone files are dominated by individuals who have done so and are selling this data. IMO, this sale of public data is unnecessary, not to mention annoying. Last year .se and .nu started making their zone files public. https://www.iis.se/english/press/pressreleases/iis-releases-zone-files/ https://www.iis.se/english/press/pressreleases/iis-releases-... https://zonedata.iis.se https://zonedata.iis.se They were also the first to try their hand at DNSSEC. As the blog post shows, DNSSEC makes it easier to get contents of a zone. FN1. "About Zone File Access Registry operators must provide to ICANN bulk access to the zone files of the Generic Top Level Domain (gTLD) at least on a daily basis. For gTLDs, a zone file contains information about domain names that are active in that gTLD. In general, Internet users may be able to access and download zone file data at no cost for certain purposes. This contractual obligation does not apply to any ccTLD (such as .us, .de, or .uk). If you have a complaint about gTLD zone file access, please submit a Zone File Access Complaint Form." source: https://www.icann.org/resources/pages/zfa-2013-06-28-en https://www.icann.org/resources/pages/zfa-2013-06-28-en
shabbyrobe 9y ago
I love "AXFR". It lets me AX FR the whole zone in one go.
- _synster_ 9y ago
  I'm surprised at the number of domains that have mis-configured Zpne Transfer even today especially given the fact the the defaults of most popular DNS servers are secure enough.
mschuster91 9y ago
CT is going to have way more impact given that people switch from paid certs to Lets Encrypt which cannot do wildcard certs. I'm looking forward to wildcard certs, anything that makes it harder for attackers helps...
- _synster_ 9y ago
  Wildcard certificates are not really a solution especially in this scenario. If you are using wildcard certs you have one more problem to deal with.
fencepost 9y ago
If you're looking for physical site ip addresses rather than dns entries, SPF records may be worth looking at (if there are distributed mail servers and outbound mail doesn't all go through a single exit point). Mail headers may be similarly informative if you can get one or more email messages out from within the office.
- _synster_ 9y ago
  Looking at SPF records is worth it. I should have added that technique.
coderholic 9y ago
Great guide. On point 6, find the asn, my own service https://ipinfo.io https://ipinfo.io can help you there. Curl ipinfo.io/ANYIP/org and you'll get the ASN info. Also see ipinfo.io/developers for more options such as geolocation, hostname, hosted domains, and more.
- _synster_ 9y ago
  Thanks. I'm the author of the blog post. I have used your service before and it has come in handy many times.
deleted 9y ago
[deleted]
Joyfield 9y ago
I have a project called DNSDigger.com that could be used as well.

308 points by adamnemecek 9y ago

Great list ! Also you can look at the SSL certificate of a page of the doamin and see if many domains appears inside it. And currently if the website uses letsencrypt and need wildcard DNS, you may find hundreds of domains. Fortunately Letsencrypt wildcard certificates come in January

nsgi 9y ago
If the website is using Let's Encrypt then any subdomains using it will be in certificate transparency logs anyway (until it supports wildcard certificates).
- nawtacawp 9y ago
  Even if they were registered/requested separately? For example, I have mail.domain.com as a separate cert than my domain.com both with letsencrypt.
  - kdbg 9y ago
    Yes, of course. The idea of certificate transparency logs is to be able to audit all of the certificates issued by a CA to detect mistakenly or maliciously issued certificates. You can go ahead and search the logs yourself through google. https://transparencyreport.google.com/https/certificates https://transparencyreport.google.com/https/certificates
- throwaway613834 9y ago
  Do you know how to find out which certificate issuers DON'T submit certs to transparency logs? Is there a list somewhere?
  - tialaramex 9y ago
    Not logging certificates will eventually cause them not to be trusted in Chrome. The deadline was pushed back but it hasn't done away. Firefox doesn't have a firm intent to do the same but it's developing all the technology. So this is probably not a good idea. Even where a subscriber specifically didn't want their certs logged and the CA doesn't currently log everything if there's ever a problem it'll get logged. This is because both Google and Mozilla will ask to see all the affected certificates, and logging them is the easiest way to satisfy this requirement. If a CA seems to be trying to dodge this usually participants will take whatever form the CA did provide and upload it to the logs. Google's logs don't care who gives them a cert, they can independently determine that it's authentic so they'll log it.
    - throwaway613834 9y ago
      Yeah, but I feel there's at least like a 30%+ chance that their attempt to make this mandatory will stop in its tracks. I just don't see corporations being okay with it.
      - jlgaddis 9y ago
        There's some work (in the CA/B?) to allow "redacting" certain information from CT logs. I don't recall all the details but I do remember reading an e-mail thread (maybe linked from HN?) about it in just the last week or so.
      - tialaramex 9y ago
        Mandatory logging is a necessary element of the CT design. What we have today is basically like the trial demo of the game. Full CT requires the completion of "gossip protocols", log auditor third parties which detect inconsistencies and a bunch of other stuff that exists today only in the laboratory, you haven't seen the half of it. The option left open by the Chrome team was what's called redaction. The logged document would be partly redacted by for example removing host names. It's not impossible that redaction will happen and end up being accepted, but its biggest proponent was Symantec, who are exiting the business after destroying trust by not exercising proper oversight on their relationship with Korea's CrossCert. And frankly the redaction proposed was getting ludicrous anyway. The trouble is that people who most want this, as with Brass Plate companies, aren't just looking to avoid excess publicity. What they want is to act entirely with impunity. And that's exactly what CT is there to prevent.

porfirium 9y ago

A small question. For a CloudFlare-protected website, I see this in crt.sh: >X509v3 Subject Alternative Name: >DNS Name:sniXXXXXXX.cloudflaressl.com (X are numbers) >DNS Name:(lots of domains, including the one I was researching) >... Does that mean all those domains belong to the same user/person?

breakingcups 9y ago
No, Cloudflare batches together many unrelated domains from different customers into a single SSL certificate.
- programd 9y ago
  Which leads to amusing issues like the following: https://arstechnica.com/information-technology/2015/03/ted-cruz-for-presidents-ssl-certificate-nigerian-prince-headache/ https://arstechnica.com/information-technology/2015/03/ted-c... Though to be fair, this incident is now fairly infamous inside Cloudflare and these days they have ways to deal with it.

dzhiurgis 9y ago

WolframAlpha can display subdomains, not sure how and the format is not easy to use. But it works. How one would one go about finding A domains pointing to specific AS. The target web app in question allows either using their subdomain or pointing to your own. Any ideas how to find those?

jcims 9y ago

The CommonCrawl dataset is another good source of host and subdomain information. The index server is here: http://index.commoncrawl.org http://index.commoncrawl.org There's a client referenced on that page. [ec2-user@x cdx-index-client]$ ./cdx-index-client.py -c CC-MAIN-2017-43 '*.ycombinator.com' 2017-11-11 22:50:08,381: [INFO]: Getting Index From http://index.commoncrawl.org/CC-MAIN-2017-43-index 2017-11-11 22:50:08,387: [INFO]: Starting new HTTP connection (1): index.commoncrawl.org 2017-11-11 22:50:08,468: [INFO]: Fetching 2 pages of *.ycombinator.com 2017-11-11 22:50:08,477: [INFO]: Starting new HTTP connection (1): index.commoncrawl.org 2017-11-11 22:50:08,478: [INFO]: Starting new HTTP connection (1): index.commoncrawl.org 2017-11-11 22:50:10,409: [INFO]: 1 page(s) of 2 finished 2017-11-11 22:50:11,711: [INFO]: 2 page(s) of 2 finished [ec2-user@x cdx-index-client]$ awk -F')' '{print $1}' domain-y* | sort | uniq -c 66 com,ycombinator 38 com,ycombinator,apply 500 com,ycombinator,blog 2 com,ycombinator,fellowship 2 com,ycombinator,macro 24036 com,ycombinator,news 25 com,ycombinator,old 2 com,ycombinator,ycblog

belorn 9y ago

For those on the defending side, remember that the purpose of DNS is to be public. Relying on DNS information to stay secret in order to protect vulnerable application is a bad idea. A service with a SQL injection is unlikely to last long, and as the article shows, it is not hard to enumerate DNS records if you are under a targeted attack.

feelin_googley 9y ago
"... remember that the purpose of DNS is to be public." DNS records, e.g. zone files, are supposed to be public information. As such, ICANN requires gTLD registries to provide users with access to them.[FN1] However, various ccTLDs at various times refused to comply, and most still try to pretend their zone files are private. Obviously, as the blog post shows, users can still get a large quantity of the data freely from other sources. Google search results for ccTLD zone files are dominated by individuals who have done so and are selling this data. IMO, this sale of public data is unnecessary, not to mention annoying. Last year .se and .nu started making their zone files public. https://www.iis.se/english/press/pressreleases/iis-releases-zone-files/ https://www.iis.se/english/press/pressreleases/iis-releases-... https://zonedata.iis.se https://zonedata.iis.se They were also the first to try their hand at DNSSEC. As the blog post shows, DNSSEC makes it easier to get contents of a zone. FN1. "About Zone File Access Registry operators must provide to ICANN bulk access to the zone files of the Generic Top Level Domain (gTLD) at least on a daily basis. For gTLDs, a zone file contains information about domain names that are active in that gTLD. In general, Internet users may be able to access and download zone file data at no cost for certain purposes. This contractual obligation does not apply to any ccTLD (such as .us, .de, or .uk). If you have a complaint about gTLD zone file access, please submit a Zone File Access Complaint Form." source: https://www.icann.org/resources/pages/zfa-2013-06-28-en https://www.icann.org/resources/pages/zfa-2013-06-28-en

shabbyrobe 9y ago

I love "AXFR". It lets me AX FR the whole zone in one go.

_synster_ 9y ago
I'm surprised at the number of domains that have mis-configured Zpne Transfer even today especially given the fact the the defaults of most popular DNS servers are secure enough.

mschuster91 9y ago

CT is going to have way more impact given that people switch from paid certs to Lets Encrypt which cannot do wildcard certs. I'm looking forward to wildcard certs, anything that makes it harder for attackers helps...

_synster_ 9y ago
Wildcard certificates are not really a solution especially in this scenario. If you are using wildcard certs you have one more problem to deal with.

fencepost 9y ago

If you're looking for physical site ip addresses rather than dns entries, SPF records may be worth looking at (if there are distributed mail servers and outbound mail doesn't all go through a single exit point). Mail headers may be similarly informative if you can get one or more email messages out from within the office.

_synster_ 9y ago
Looking at SPF records is worth it. I should have added that technique.

coderholic 9y ago

Great guide. On point 6, find the asn, my own service https://ipinfo.io https://ipinfo.io can help you there. Curl ipinfo.io/ANYIP/org and you'll get the ASN info. Also see ipinfo.io/developers for more options such as geolocation, hostname, hosted domains, and more.

_synster_ 9y ago
Thanks. I'm the author of the blog post. I have used your service before and it has come in handy many times.

deleted 9y ago

[deleted]

Joyfield 9y ago

I have a project called DNSDigger.com that could be used as well.