It's just that people are suspicious of unknown crawlers, and rightly so. Since it is impossible to know a priori which crawler are malicious, and many are mal

It's just that people are suspicious of unknown crawlers, and rightly so. Since it is impossible to know a priori which crawler are malicious, and many are malicious, it is reasonable to default to considering anything unknown malicious.