5 ms·
This is what made me laugh: The Administrative Office of the U.S. Courts reported that the PACER system was being inundated with requests. One request was bein
by NathanKP 17y ago
This is what made me laugh:
The Administrative Office of the U.S. Courts reported that the PACER system was being inundated with requests. One request was being made every three seconds.
Inundated by one request every three seconds? That's pretty pathetic if you ask me. They must not have planned for the PACER system to be used very much by very many people.
- riffer 17y agoThat's fair, but remember that if you scrape HN with a 'sleep 3' your IP gets throttled quickly and then eventually blocked ...
- markerdmann 17y agoWhat would be an appropriate delay for HN?
- harpastum 17y agoIt depends on what you're using it for. Unless you need the bleeding edge of new stories, 30-45 minutes between each scrape is probably fine. If you're looking to stay on top of 'new', your best bet is about every 10 minutes. More frequent than that, and you're just causing trouble for the server with no real benefit for yourself.
- mahmud 17y agoPaul already provides data dumps. Searchyc for it.
- NathanKP 17y agoI wonder if that is pretty standard? Do most services throttle at 3 seconds or is that being overly strict?
- kevindication 17y agoMost are only throttled by the bandwidth at either end.