Firstly, read the robots.txt page (https://news.ycombinator.com/robots.txt https://news.ycombinator.com/robots.txt), it states a crawl-delay of 30 seconds so do

Firstly, read the robots.txt page (https://news.ycombinator.com/robots.txt https://news.ycombinator.com/robots.txt), it states a crawl-delay of 30 seconds so don't crawl faster than that. Secondly, obey the robots.txt page. Don't crawl pages listed in their Disallow rules.