Concerning the crawling aspect of this, Michael Nielsen wrote a great article about how to download/crawl a significant part of the Internet in 40 hours: http:

Concerning the crawling aspect of this, Michael Nielsen wrote a great article about how to download/crawl a significant part of the Internet in 40 hours: http://www.michaelnielsen.org/ddi/how-to-crawl-a-quarter-billion-webpages-in-40-hours/ http://www.michaelnielsen.org/ddi/how-to-crawl-a-quarter-bil... It's a bit dated (from 2012) but probably still relevant.