Problem is, that by disallowing those smaller players, you are preventing them from getting bigger because they would need to be able to index sites to actually

Problem is, that by disallowing those smaller players, you are preventing them from getting bigger because they would need to be able to index sites to actually become good enough to attract visitors. Also, when disallowing a spider you can be sure that you won't get ANY visitors from them, as you are not in their index to start with. I mean: if I disallow any bot but Googlebot, I will have 100% Google referrals by design. I can understand that in an emergency situation you might want to block some bots, but blocking all of them because they are currently small feels a) a bit unfair due to not giving them the chance to grow and b) shortsighted due to never knowing if they might be getting big enough or the visitors they deliver might be better ad-clickers