> Webmasters also put highly specific URLs they don't want the world to see in robots.txt. Also not a great idea. I can confirm; I tried to helpfully put a bun

> Webmasters also put highly specific URLs they don't want the world to see in robots.txt. Also not a great idea. I can confirm; I tried to helpfully put a bunch of addresses which would be pointless for a web crawler to crawl in there (links to the login page, etc), but as soon as you say "don't" somebody's gotta go find out why. I like the idea of putting gzip bombs at commonly-scanned URLs.