Another user posted a similar question a few days ago asking if it would be alright if he crawled HN, and would Paul block him if he attempted to do so. I told

Another user posted a similar question a few days ago asking if it would be alright if he crawled HN, and would Paul block him if he attempted to do so. I told him: "There's enough of a load on HN as it is without every Tom, Dick, and Harry testing out his proof of concept by crawling this site." I think a year ago this wouldn't have been an issue if you spaced your requests out by several seconds, but during the daytime (United States daytime, that is) this site is hammered. Yesterday the site was brought to a crawl (no pun intended) by just from everyone commenting on that DDG/Google article. If you spaced your requests out by 3 or 4 seconds and only crawled during the US night time then the servers shouldn't be affected too much. That doesn't account for the data transfer you use up though. Someone has to pay for that and there are already several sites (that I can think of off the top of my head) that crawl HN on a regular basis.