3 ms·
Googlebot is pretty careful and generally doesn’t cause these problems.
by dodobirdlord 6y ago
Googlebot is pretty careful and generally doesn’t cause these problems.
- spijdar 6y agoRight, then they shouldn't be effected by the rate-limiting, as long as its reasonable. If it was applied evenly to all clients/crawlers, it'd at least allow the possibility for a respectful, well designed crawler to compete.
- jedberg 6y agoThe problem is, if you own a website, it takes the same amount of resources to handle the crawl from Google and FooCrawler even if both are behaving, but I'm going to get a lot more ROI out of letting Google crawl, so I'm incentivized to block FooCrawler but not Google. In fact, the ROI from Google is so high I'm incentivized to devote extra resources just for them to crawl faster.
- progval 6y agoWe know that. No one claims websites are doing this for no reason. It's explicitly written in the article. But this sub-thread is about misbehaved crawlers.