4 ms·
That's fair, but remember that if you scrape HN with a 'sleep 3' your IP gets throttled quickly and then eventually blocked ...
by riffer 17y ago
That's fair, but remember that if you scrape HN with a 'sleep 3' your IP gets throttled quickly and then eventually blocked ...
- markerdmann 17y agoWhat would be an appropriate delay for HN?
- harpastum 17y agoIt depends on what you're using it for. Unless you need the bleeding edge of new stories, 30-45 minutes between each scrape is probably fine. If you're looking to stay on top of 'new', your best bet is about every 10 minutes. More frequent than that, and you're just causing trouble for the server with no real benefit for yourself.
- mahmud 17y agoPaul already provides data dumps. Searchyc for it.
- NathanKP 17y agoI wonder if that is pretty standard? Do most services throttle at 3 seconds or is that being overly strict?
- kevindication 17y agoMost are only throttled by the bandwidth at either end.