> 2) You don't have to respect robots.txt. If you need to crawl a 1 million page site, and its robots.txt restricts you to 1 page/second, then you'll have to wa

> 2) You don't have to respect robots.txt. If you need to crawl a 1 million page site, and its robots.txt restricts you to 1 page/second, then you'll have to wait for a long time. Yeah, consider a site like Hacker News, where the crawl delay is not 1 second but 30 seconds [1]. If you're trying to grab historical data like iHackerNews did [2], you might be better off hitting Google's webcache instead...except, to enumerate urls you have to scrape the search results pages, AFAIK. This is why Google opening its webcache via an API is a GREAT idea. [1] http://news.ycombinator.com/robots.txt http://news.ycombinator.com/robots.txt [2] http://api.ihackernews.com/ http://api.ihackernews.com/