3 ms·
If all scrapers did what you did, I'd curse a lot less at $work. Kudos for that. Re 2 and 3: do you parse/respect the "Crawl-delay" robots.txt directive, and d
by mfontani 7y ago
If all scrapers did what you did, I'd curse a lot less at $work. Kudos for that.
Re 2 and 3: do you parse/respect the "Crawl-delay" robots.txt directive, and do you ensure that works properly across your fleet of crawlers?
- the8472 7y agoIn addition to crawl-delay there's also HTTP 429 and the retry-after header. https://tools.ietf.org/html/rfc6585#page-3 https://tools.ietf.org/html/rfc6585#page-3
- greglindahl 7y agoSites also use 403 and 503 to send rate-limit signals, despite what the RFCs say.
- aspyct 7y agoHehe, my "fleet of crawlers" is a single machine in a closet so far :) I'll think about that kind of synchronization later. However I do parse and respect the "crawl-delay" now, thanks for pointing it out!
- greglindahl 7y agoA large fraction of websites with Crawl-Delay set it a decade ago and promptly forgot about it. No modern crawler uses it for anything other than a hint. The primary factors for crawl rate are usually site page count and response time.