2 ms·
A notable one: http://hacker-newspaper.gilesb.com/ http://hacker-newspaper.gilesb.com/ Has PG ever discouraged or prevented HNers from creating such tools? Th
by petercooper 16y ago
A notable one: http://hacker-newspaper.gilesb.com/ http://hacker-newspaper.gilesb.com/
Has PG ever discouraged or prevented HNers from creating such tools?
The only thing that I recall was discouraged was a Greasemonkey script that resulted in HN getting hammered with lots of requests on every page (I forget what it did but it did a lot of lookups on HN). People who used it had their IPs banned for a day or so.
- SudarshanP 16y agohttp://hacker-newspaper.gilesb.com/ http://hacker-newspaper.gilesb.com/ is awsome!!! I am planning to crawl HN using http://scrapy.org http://scrapy.org Strictly adhering to http://news.ycombinator.com/robots.txt http://news.ycombinator.com/robots.txt will severely limit the usefulness of the data scraped. User-Agent: * Disallow: /x? Disallow: /vote? Disallow: /reply? Disallow: /submitted? Disallow: /threads? So I hope to crawl HN /x,/threads etc. Of course I plan to do it as slowly and unobtrusively as possible. If a database dump or archive is available, the whole crawling could be skipped.