Partial plug, but very related to the topic: if you're doing large scale analysis on the web and you don't want to have to actually run a large scale crawl, use

Partial plug, but very related to the topic: if you're doing large scale analysis on the web and you don't want to have to actually run a large scale crawl, use the CommonCrawl dataset[1]! Common Crawl is a non profit organization that wants to allow anyone to use big web data. I'm one of the team behind the crawl itself. Last month (July) we downloaded 4 billion web pages. Thanks to Amazon Public Datasets, all of that data is freely distributed via Amazon S3, under a very permissive license (i.e. good for academics, start-ups, businesses, and hobbyists). If your hardware lives on EC2, you can process the entire thing quickly for free. If you have your own cluster and many many terabytes of storage, you can download it too! People have used the dataset to generate hyperlink graphs[2], web table content[2], microdata[2], n-gram and language model data (ala Google N-grams)[3], NLP research on word vectors[4], and so on, so there's a lot that can be done! [1]: http://commoncrawl.org/ http://commoncrawl.org/ [2]: http://webdatacommons.org/ http://webdatacommons.org/ [3]: http://statmt.org/ngrams http://statmt.org/ngrams [4]: http://nlp.stanford.edu/projects/glove/ http://nlp.stanford.edu/projects/glove/