4 ms·
It seems like threads only become paginated when they become closed for comments. I think this makes sense in some ways. I noticed it because I way testing, wen
by etcet 15y ago
It seems like threads only become paginated when they become closed for comments. I think this makes sense in some ways. I noticed it because I way testing, went to sleep, woke up and refreshed the thread I was testing with and everything broke.
Your approach is what I was thinking of doing. It's good to hear that it's working for you. Do you perform any queueing or waiting? I was worried that the reportedly aggressive rate limiter might block someone who opened a few large threads at once.
- chaosprophet 15y agoI'm not really using it to scrape the comments threads in bulk, I'm just using it on an individual thread at a time. So I haven't had any problems with rate limiting. Also, unless you are hitting hundreds of threads at the same time, I don't really think you will be caught by the rate limiter. Not sure about this though, but it's something that has been discussed before on HN. Try searching for something like 'HN crawl rate', and you will get more info.