I'm not really using it to scrape the comments threads in bulk, I'm just using it on an individual thread at a time. So I haven't had any problems with rate lim

I'm not really using it to scrape the comments threads in bulk, I'm just using it on an individual thread at a time. So I haven't had any problems with rate limiting. Also, unless you are hitting hundreds of threads at the same time, I don't really think you will be caught by the rate limiter. Not sure about this though, but it's something that has been discussed before on HN. Try searching for something like 'HN crawl rate', and you will get more info.