3 ms·
Don't forget the Crawl-delay on HN's robots.txt: http://news.ycombinator.com/robots.txt http://news.ycombinator.com/robots.txt
by DanielRibeiro 15y ago
Don't forget the Crawl-delay on HN's robots.txt: http://news.ycombinator.com/robots.txt http://news.ycombinator.com/robots.txt
- mvanveen 15y agoIt should only scrape the front page once per run as an initial condition to unleash the other requests. But thanks for the heads up! I'll be sure to mention this in the docs. Edit: changes have been posted. Thanks for the suggestion!