Be careful not to hammer the site. Your IP could be added to the blocklist if you are too aggressive: "Yes, we block IPs that seem to be crawlers ignoring robo

Be careful not to hammer the site. Your IP could be added to the blocklist if you are too aggressive: "Yes, we block IPs that seem to be crawlers ignoring robots.txt. We've always blocked abusive IPs, but I tightened up the blocking a few weeks ago. A lot of people were crawling HN, most of them unnecessarily because they were doing things they could have done more efficiently through HNSearch's API[1]." --pg[2] [1] http://www.hnsearch.com/api http://www.hnsearch.com/api [2] http://news.ycombinator.com/item?id=3196298 http://news.ycombinator.com/item?id=3196298