Don't bother; robots.txt is utterly useless. It is widely ignored by bots. IDEA: What you could do is instantly ban anything with "bot" in its agent string, wh

Don't bother; robots.txt is utterly useless. It is widely ignored by bots. IDEA: What you could do is instantly ban anything with "bot" in its agent string, which accesses your site without having probed robots.txt first. That's how robots.txt could be useful, whether you have one or not. Anything which doesn't even bother fetching it is rude garbage.