3 ms·

I agree wholeheartedly. For comparison, I currently see ~95% of search engine referral traffic from Google, 2.8% from Yahoo, 1.4% from Bing, and <1% from all ot

by bobf 16y ago

I agree wholeheartedly. For comparison, I currently see ~95% of search engine referral traffic from Google, 2.8% from Yahoo, 1.4% from Bing, and <1% from all other search engines combined. It makes no sense to let X, Y, Z random search engines index content like Google, when they send nowhere near the traffic. As others have mentioned, blocking by robots.txt often doesn't work as some bots don't obey it. I currently use an extensive rewrite rule to block those bots' User-Agents, and block specific IPs as a last resort.

pilif 16y ago
Problem is, that by disallowing those smaller players, you are preventing them from getting bigger because they would need to be able to index sites to actually become good enough to attract visitors. Also, when disallowing a spider you can be sure that you won't get ANY visitors from them, as you are not in their index to start with. I mean: if I disallow any bot but Googlebot, I will have 100% Google referrals by design. I can understand that in an emergency situation you might want to block some bots, but blocking all of them because they are currently small feels a) a bit unfair due to not giving them the chance to grow and b) shortsighted due to never knowing if they might be getting big enough or the visitors they deliver might be better ad-clickers