3 ms·
Nothing at all so far. I've been pretty careful to be polite though - the majority of my scrapers work no more than once an hour, and the few I've done that ar
by simonw 5y ago
Nothing at all so far.
I've been pretty careful to be polite though - the majority of my scrapers work no more than once an hour, and the few I've done that are more frequent than that don't run more than once every ten minutes (GitHub Actions can't be scheduled more often than that).
Those tend to be sites which I would expect to be able to handle the traffic very easily.
When I'm operating sites myself I don't tend to even notice a scraper that only hits a single page every few minutes. The things that hurt me are crawlers that hit every page on a site, potentially more than a request per second and with traffic patterns that are certain to hit un-cached sources.
- JNRowe 5y agoExcellent, thanks! I'm purposely being careful too. Big backoff on any error, curl's --time-cond from git commit timestamp, etag where possible, etc. I'm not particularly concerned about an actual DoS'ing issue, more being on the end of one of those stories that pops up every so often("website owner sics lawyers on F12").
- simonw 5y agoOh interesting - I hadn't considered using --time-cond. I'm always a bit suspicious of If-Modified-Since because I'm never sure which timezones will be in effect - so I tend to use ETags instead, which achieve the same goal without involving any clock time. I actually wrote a tiny Python CLI tool for doing those, designed to be used in CI: https://pypi.org/project/conditional-get/ https://pypi.org/project/conditional-get/