3 ms·
The CommonCrawl dataset is another good source of host and subdomain information. The index server is here: http://index.commoncrawl.org http://index.commoncra
by jcims 9y ago
The CommonCrawl dataset is another good source of host and subdomain information. The index server is here: http://index.commoncrawl.org http://index.commoncrawl.org There's a client referenced on that page.
[ec2-user@x cdx-index-client]$ ./cdx-index-client.py -c CC-MAIN-2017-43 '*.ycombinator.com'
2017-11-11 22:50:08,381: [INFO]: Getting Index From http://index.commoncrawl.org/CC-MAIN-2017-43-index
2017-11-11 22:50:08,387: [INFO]: Starting new HTTP connection (1): index.commoncrawl.org
2017-11-11 22:50:08,468: [INFO]: Fetching 2 pages of *.ycombinator.com
2017-11-11 22:50:08,477: [INFO]: Starting new HTTP connection (1): index.commoncrawl.org
2017-11-11 22:50:08,478: [INFO]: Starting new HTTP connection (1): index.commoncrawl.org
2017-11-11 22:50:10,409: [INFO]: 1 page(s) of 2 finished
2017-11-11 22:50:11,711: [INFO]: 2 page(s) of 2 finished
[ec2-user@x cdx-index-client]$ awk -F')' '{print $1}' domain-y* | sort | uniq -c
66 com,ycombinator
38 com,ycombinator,apply
500 com,ycombinator,blog
2 com,ycombinator,fellowship
2 com,ycombinator,macro
24036 com,ycombinator,news
25 com,ycombinator,old
2 com,ycombinator,ycblog