5 ms·
Search engine crawlers use adaptive politeness: start being very polite, and ramp up parallel fetches if the site responds quickly and has a lot of pages.
by greglindahl 10y ago
Search engine crawlers use adaptive politeness: start being very polite, and ramp up parallel fetches if the site responds quickly and has a lot of pages.
- stummjr 10y agoThat's kind of what Scrapy's AUTO_THROTTLE middleware does.