I've used it and it's great. However, many sites disallow all crawling via robots.txt and only whitelist certain crawlers such as Google or Bing. These sites wi

I've used it and it's great. However, many sites disallow all crawling via robots.txt and only whitelist certain crawlers such as Google or Bing. These sites will be "missing" from CommonCrawl. One thing CommonCrawl really needs is a good index. https://github.com/trivio/common_crawl_index https://github.com/trivio/common_crawl_index is great, but it only exists for the data from 2012. Apparently there was something in the works for 2014 data but there hasn't been any update for months: https://groups.google.com/forum/#!topic/common-crawl/mrZBnvDvVvo https://groups.google.com/forum/#!topic/common-crawl/mrZBnvD...