Whitelisting a couple of bots now doesn't help at all for any new search engines trying to start up. What are they to do, contact every site admin individually?

Whitelisting a couple of bots now doesn't help at all for any new search engines trying to start up. What are they to do, contact every site admin individually? > if some new bot were to misbehave and crawl too aggressively, blacklisting would only help after the fact. In that case, don't blacklist all bots, simply add a crawl delay to any bots that you haven't specifically allowed: User-agent: * Crawl-delay: 10 This allows minor bots to continue to crawl the site, while cutting back on bandwidth costs for the couple of ones that are being overly aggressive.