You don't need to crawl the web yourself. http://commoncrawl.org/ http://commoncrawl.org/ will give you the data for free. It's a little out of date by the time

You don't need to crawl the web yourself. http://commoncrawl.org/ http://commoncrawl.org/ will give you the data for free. It's a little out of date by the time you get it but that shouldn't effect a project like this.