3 ms·
Search engine crawlers are a subset of crawlers. Sometimes you’re dealing with aggressive screen-scraping from competitors or various marketing tools. I’ve had
by kayodelycaon 3y ago
Search engine crawlers are a subset of crawlers. Sometimes you’re dealing with aggressive screen-scraping from competitors or various marketing tools.
I’ve had to deal with these things easily bringing sites down.
- tracker1 3y agoSame, and when more than half your links are dynamic search results, it can pile on and really bring things to a crawl. I worked on a fairly popular auto classifieds website, and more than 90% of traffic was various scrapers, and some were definitely a burden. Worse, is that it doesn't show up in analytics as it's not running client-js... Ironically equally bad was when bing started scraping with JS and it skewed google analytics. If all we had to deal with were the users, wouldn't need nearly the spend on the site. Started manually blocking some of the worst offenders.