4 ms·
In my experience the best way to crawl in a polite way is to never use an asynchronous crawler. The vast majority of small to medium sites out there have absolu
by elorant 10y ago
In my experience the best way to crawl in a polite way is to never use an asynchronous crawler. The vast majority of small to medium sites out there have absolutely no kind of protection from an aggressive crawler. You make 50 to 100 requests per second chances are you’re DDoS-ing the shit out of most sites.
As for robots.txt problem is most sites don’t even have one. Especially e-commerce sites. They also don’t have a sitemap.xml in case you don’t want to hit every url just to find the structure of the site. Being polite in many cases takes a considerable effort.
- jsargiox 10y agoI agree, at the end of the day being polite or not is on the developer and not the tool itself...
- stummjr 10y agoScrapy is asynchronous, but it provides many settings that you can use to avoid DDoS a website, such as limiting the amount of simultaneous requests for each domain or IP address. And yes, crawling politely requires a bit of effort from both ends: the crawler and the website.
- zeroxfe 10y agoYou can rate-limit asynchronous crawlers too.
- greglindahl 10y agoSearch engine crawlers use adaptive politeness: start being very polite, and ramp up parallel fetches if the site responds quickly and has a lot of pages.
- stummjr 10y agoThat's kind of what Scrapy's AUTO_THROTTLE middleware does.