http://hacker-newspaper.gilesb.com/ http://hacker-newspaper.gilesb.com/ is awsome!!! I am planning to crawl HN using http://scrapy.org http://scrapy.org Strict

http://hacker-newspaper.gilesb.com/ http://hacker-newspaper.gilesb.com/ is awsome!!! I am planning to crawl HN using http://scrapy.org http://scrapy.org Strictly adhering to http://news.ycombinator.com/robots.txt http://news.ycombinator.com/robots.txt will severely limit the usefulness of the data scraped. User-Agent: * Disallow: /x? Disallow: /vote? Disallow: /reply? Disallow: /submitted? Disallow: /threads? So I hope to crawl HN /x,/threads etc. Of course I plan to do it as slowly and unobtrusively as possible. If a database dump or archive is available, the whole crawling could be skipped.