Fair enough. I've always seen my robots.txt files as a friendly way of saying "you probably don't need to worry about this stuff" to crawlers. That's not your

Fair enough. I've always seen my robots.txt files as a friendly way of saying "you probably don't need to worry about this stuff" to crawlers. That's not your use case, and I'm happy to have my "Disallowed" content to be crawled, I expect poorly written bots to do so anyway. If I actually want to prevent browsing/crawling of something, I wont ask politely I'm just going to throw up 403s.