2 ms·
The problem is, if you own a website, it takes the same amount of resources to handle the crawl from Google and FooCrawler even if both are behaving, but I'm go
by jedberg 6y ago
The problem is, if you own a website, it takes the same amount of resources to handle the crawl from Google and FooCrawler even if both are behaving, but I'm going to get a lot more ROI out of letting Google crawl, so I'm incentivized to block FooCrawler but not Google. In fact, the ROI from Google is so high I'm incentivized to devote extra resources just for them to crawl faster.
- progval 6y agoWe know that. No one claims websites are doing this for no reason. It's explicitly written in the article. But this sub-thread is about misbehaved crawlers.