3 ms·
I wonder if anyone has processed Common Crawl's dataset for a list of all the existing domains (I don't care if the data are old).
by elorant 9y ago
I wonder if anyone has processed Common Crawl's dataset for a list of all the existing domains (I don't care if the data are old).
- RobAley 9y agoYou can get the URL data (and parse out the domains yourself) at http://index.commoncrawl.org/ http://index.commoncrawl.org/ Be aware that commoncrawl, while huge, only crawls a fraction of the web, and so you won't get all of the "existing domains" (i.e. all domains that exist) if that is what you are after.