I've seen several blog posts on exceptional cases before, but the only case I've personally witnessed is how the crawl-delay directive is ignored. Granted you c

I've seen several blog posts on exceptional cases before, but the only case I've personally witnessed is how the crawl-delay directive is ignored. Granted you can set it if you're a webmaster tools user, but even then the setting disables itself after a few months.