3 ms·
On the other hand, I do not want my site to go down thanks to a few bad 'crawlers' that fork() a thousand http requests every second and take down my site, forc
by judge2020 6y ago
On the other hand, I do not want my site to go down thanks to a few bad 'crawlers' that fork() a thousand http requests every second and take down my site, forcing me to do manual blocking or pay for a bigger server/scale-out my infrastructure. Why should I have to serve them?
- progval 6y agoYou can use the same rate-limiting for all crawlers, Google or not.
- dodobirdlord 6y agoGooglebot is pretty careful and generally doesn’t cause these problems.
- spijdar 6y agoRight, then they shouldn't be effected by the rate-limiting, as long as its reasonable. If it was applied evenly to all clients/crawlers, it'd at least allow the possibility for a respectful, well designed crawler to compete.
- jedberg 6y agoThe problem is, if you own a website, it takes the same amount of resources to handle the crawl from Google and FooCrawler even if both are behaving, but I'm going to get a lot more ROI out of letting Google crawl, so I'm incentivized to block FooCrawler but not Google. In fact, the ROI from Google is so high I'm incentivized to devote extra resources just for them to crawl faster.
- progval 6y agoWe know that. No one claims websites are doing this for no reason. It's explicitly written in the article. But this sub-thread is about misbehaved crawlers.
- Exuma 6y agoAgree... this entire argument seems to think I give a rats ass about 9000 different crawlers that give me literally zero benefit and only waste server resources. Most of those crawlers are for ad-soaked piss poor search engines. I'd rather just block them all and allow crawlers that don't know how to behave.
- deleted 6y ago[deleted]