> Specifically: why do so many people ban web crawlers other than the "large" ones, Never heard of this. Third party crawlers usually get banned because they a

> Specifically: why do so many people ban web crawlers other than the "large" ones, Never heard of this. Third party crawlers usually get banned because they are poorly made (get stuck) or disobey robots.txt.